How to Run a Great iPhone Beta Test

The App Store has over 100,000 apps available for purchase.

That’s a big number.

I can’t tell you the secret to success in the App Store. I haven’t discovered any magic formulas. The one thing I can tell you with 100% certainty is this: if you ship a product with bugs that genuinely piss off your customers or, worse, make them unable to use some portion of your hard work, the road toward success just got much longer and a lot more miserable. Even with “prioritized review” requests, once your buggy app is on the store, the review queue means you could be waiting a long time before your customers can enjoy your bug fix.

You will not sleep well at night.

The good news is that this problem has been solved a thousand times over. A thorough beta test can save you from this terrible outcome and the sleepless nights that come with it. With a bit of diligence, you can ship an app with awesome stability to match your great ideas.

I’ll share how I run my betas. This is about process — the technical side of Ad Hoc distribution is thoroughly documented by Apple and others. The tools I use are simple and free. They work. If you have suggestions for better ones, I want to know about them before my next test! Please share in the comments.

Gather Your Testers

You’re going to need testers. If you use a volunteer testing pool, figure that 80% of testing will be done by 20% of your volunteers. That means to have any hope of gathering meaningful feedback, you’ll need at least 10 volunteers. 15 or 20 would be the ideal minimum.

Next, you’ll need to start early — but not so early that your testers’ enthusiasm cools by the time you ship your first beta build. Try to get the word out between three to six weeks before you’re ready to test. If you have a particularly strong network, you may need less time. If you’re relying more on the kindness of strangers, though, make sure to give yourself plenty of lead time. Use your website, your blog and your Twitter to solicit volunteers but don’t be a pest.

Most importantly: if you have contact with existing customers from previous versions or perhaps other products entirely, send a quick, polite email explaining what you’re going to test and encouraging them to join up. Customers who have emailed you in the past with feedback or feature requests may be especially keen on seeing your latest work.

Understand Your Testing Pool

You need to know things about your testers. What kind of hardware will they be using? What version of iPhone OS? What’s their geographic region? (International testers are crucial for finding your screwups with localized formatting and other geographic issues.) Most importantly, you’re going to need to collect their UUID.

I use Forms for Google Docs to collect most of my feedback. It’s an easy-to-configure way to store structured data. Since you can designate required fields, you can ensure your testers don’t forget to include anything crucial when reporting issues. Best of all, you can embed the forms into existing web pages, allowing you to include them alongside testing guidance, version history or other information that might be useful.

Here’s the form I use to collect beta tester info:

It’s not short. The other benefit to using a form is to ensure you only get testers who are reasonably skilled at providing detailed information. Everyone who makes it past this filter has a decent shot at actually telling you everything you need to know later in the beta.

Betas Worth Testing

If your app doesn’t look somewhat close to the final product, don’t distribute it for testing. If you’re relying on a volunteer testing pool, you want them eager to use your stuff. If your overall UI is incomplete, you’re not ready to test. You’ve got a limited amount of time and attention with your testers. Don’t squander it on early builds you could easily test yourself. The closer you think your app is to shipping, the more useful your testers will be to you.

You do yourself no favors if the last bit of functionality you add affects previously-solid features that now need a whole new round of testing.

Believe me.

Distributing Your Binaries

So you’ve toiled away and you’ve finally got an app you feel is ready to be tested.

Don’t email the binary.

Depending on the size of your app, the attachment may not be properly delivered to the recipient, especially if they’re using Exchange-hosted email. Instead, host your app and the necessary provisioning profile on the web.

I also recommend a simple checkout system. When you’ve got a new build ready for testing, send your testers a link to a one-question form that asks for their email address. On the form confirmation screen, you can provide a link to the binary. This lets you get an idea of who is participating. It also lets you answer an important question: am I getting zero feedback because there are no bugs or because there are no active testers? With a checkout process, you’ll know for sure.

Use clear versioning with every distributed build. You need to be certain you and your users are talking about the same build when they give you feedback. Beta phases, version numbers, dates, whatever you need to make things clear for everyone involved. Make sure that somewhere in your app, your code pulls the version number out of your app’s Info.plist file and displays it in the UI. Here’s some code to get that string:

NSString *currentVersion = [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleVersion"];

Update your Info.plist’s version number religiously before distribution. Even better, update it automatically.

Communication

You need to communicate with your testing pool. Keeping them regularly updated, without being spammy, is crucial to keeping them excited about what’s going on.

When you send out a new build, explain what’s new. Give specific testing guidance so that your testers know where they should focus their efforts. New method of rendering cells you strongly suspect has a bug or two? Share your hunches. Built a new import subsystem you’re suspicious might break with legacy data? Provide an example data set or encourage existing users of the old version to give it a whirl. If you’re manipulating data that lives only on the user’s device, tell them to back up their iPhone before running your beta. And tell them why. Don’t be a douche on this: never take for granted the importance of your testers’ application data, especially if you’re lucky enough to have your existing customers helping you. Yes, we all know, betas are risky. Still, the iPhone isn’t the desktop and if you’re testing an upgrade to an existing app, your customers may have data they created before accepting the risks of a beta.

If it has been more than seven to ten days since your last build, send a quick update to the pool. Call out awesome testers and let them know they’re making a difference. Share the score of resolved issues you were able to fix thanks to the pool’s feedback. Most of all, thank them. Their efforts are helping you to not look like an idiot.

Sandboxing Helps

If you’re making an all-new app, you can skip this section. If you’re testing an upgrade to an existing app, read on.

Your early beta builds should use a different app identifier than your shipping app. This way, your beta will be installed as a distinct application alongside your currently shipping version on the user’s device, instead of overwriting it. This lets you ensure that no shenanigans happen to legacy data if you’ve got customers from previous versions in your testing pool. Toward the end of your beta, after much internal testing, you can start distributing builds that overwrite previous shipping versions. You’ll want to ensure that the app properly imports legacy data and preferences and, of course, ensure nothing otherwise disastrous happens when iTunes distributes your updated app to existing customers with existing data.

Gather Useful Feedback

You need to know things about how your application is failing. The forms I use are inspired heavily by what I saw once upon a time during the iPhone OS 2.0 Enterprise AppleSeed beta test. I think it covers the basics. Of course, don’t be afraid to add other questions specific to your application’s circumstances. For bugs, measure how well you can reproduce the bug and its overall severity while capturing the reproduction steps. For feature or enhancement requests, measure the tester’s impression of the request’s importance along with what they’re asking for.

However you gather your feedback, ensure that every single message you send to your testing pool includes brief instructions on how your testers should report their bugs. Never make them hunt around for it.

Ship

You haven’t added or changed anything for a couple of builds. You’ve fixed plenty of bugs and no one has reported regressions. You’ve sent emails to your top-performing testers and they’re giving you a thumbs-up.

Give it one last test yourself. Thirty minutes, minimum, actually doing things your users are likely to do. Is it kosher?

Then ship it.

Then, while you wait for approval from Apple, test it some more, informally, yourself. If somehow a show-stopping bug pops up while you await approval, you can reject your binary and submit a fix before it ruins anyone’s day.

One of my proudest managed products: Turkey Tacos

I’ve had the product management bug since the early days of my childhood. While I have a stable of iPhone products that I spend the bulk of my time planning and developing, my need to make good stuff doesn’t end with software.

It extends entirely into the kitchen. I love cooking. It’s a creative act whose rewards are unrivaled by anything else. When you get it right, when the flavors and textures come together in a way that can satisfy and surprise, it feels incredible. Moreover, cooking is an important bit of empowerment. In today’s world of corporate food, unless you prepare it yourself, you’ll have no idea what’s truly happening on your plate. Cooking your own meals is the best way to understand your diet and cut out synthetic, unhealthy garbage. 90% of my meals come from my own kitchen prepared with ingredients that are as elemental as possible.

Cooking for me breaks down to about 25% experimentation and 75% refinement. That is, while I often try new things, I spend most of my time taking my established specialties and making them better. Of all of my kitchen products thus managed, I’m proudest of Turkey Tacos.

I love tacos: crunchy, savory and infinitely customizable. My favorite taco was always ground beef, but it had some drawbacks: greasy as hell yet the meat is paradoxically dry and crumbly. I don’t love ground turkey for many of the applications where it is recommended. Turkey burgers are odd, spongey, colorless things and turkey meatballs suffer from similar drawbacks.

Yet, for tacos, ground turkey is sublime. The meat breaks up into manageable, small chunks while remaining superbly moist and tender. The sponginess that is an indictment against turkey burgers is an asset for tacos, as the meat even after browning remains bursting with flavor.

The best part of turkey tacos: economy. They’re inexpensive to make and the prep time is less than 20 minutes.

Here’s my current, favorite formulation after over a dozen iterations. No sodium bomb flavor packet full of garbage, here. Just a tasty meal with ingredients you’ll look straight in the eye before cooking.

Turkey Tacos

You will need:

• Half a pound of ground turkey
• A can of simple, smooth tomato sauce (or, if you make yours from scratch, about a cup)
• A 4oz can of Green Chile (diced green chile pepper is a delicious addition to any meal. You can usually find it among your other canned foods at the Supermarket or at Trader Joes. Sometimes it’s hiding among the ethnic foods. Use whatever temperature suits you. Of course, if you’re lucky enough to live in New Mexico, substitute canned for fresh)
• A few spoonfuls of Frontera Chipotle salsa (can substitute with another splash of tomato sauce and some crushed red pepper, but seriously, find this salsa — it’s incredible)
• Olive oil
• Salt
• Black pepper
• Two cloves of garlic, minced or crushed
• Blue corn taco shells (I’ve found that Whole Foods’ house brand is the best of these and costs the same as nasty garbage from Old El Paso)
• Shredded cheese
• Optional: Frontera Chipotle Garlic Taco Sauce. Absurdly good as a topping at the end

Preparation:

In a skillet or electric wok, heat two tablespoons of olive oil over medium heat, then add your garlic. Keep the garlic moving so it doesn’t burn. Once the garlic has gone from white to light brown, carefully add the turkey. Try to avoid pouring in any of the water or other fluid that your turkey may be sitting in — your olive oil will spatter otherwise. Break up the turkey into small chunks while you brown it, again keeping it moving to prevent burning. As the turkey browns, add a light sprinkle of salt and pepper, according to your preferences. Better to be conservative at this stage — you want just enough to lightly flavor the meat and you can always add more later before you serve.

Take moment to preheat your oven according to the directions on your taco shells.

Once the meat is evenly brown, with nothing pink left in your pan, add the green chiles. Since turkey is so much less fatty, you won’t need to drain the pan of grease as you would with ground beef at this stage. If anything, your pan may be getting a little dry and sticky, especially if it doesn’t have non-stick coating. Add a splash of olive oil if necessary to keep things lubricated. After about a minute, once the green chile is cooked through, stir in the tomato sauce.

After everything has been combined and the meat has a light, even coat of tomato sauce, turn off the heat and have a taste of your taco filling. If you’re satisfied with the spiciness and overall flavor, great, you’re all set. If you want to ratchet up the heat, though, add a few spoonfuls of the Chipotle Salsa, then taste again. Repeat until it’s just as you like it.

Let the filling rest and cool while you toast the taco shells according to the directions on their box. Usually about a minute at 400 degrees F. While the shells cook, transfer your taco filling to a serving bowl. Set the table with your bowl of turkey, the cheese, your taco sauce and the salsa. Bring the shells to the table, then let the table take turns building tacos. Two or more spoons in the turkey bowl are a good idea. Fill with turkey, top with sauce, salsa and cheese, to preferences. Makes about eight tacos.

A couple of notes: Like I said, I try to build with the most elemental, simple ingredients possible. At the same time, the Frontera Taco Sauce and Salsa are outstanding shortcuts. Short list of natural ingredients, made in small batches, no scary garbage and really, really tasty. Besides all that, Rick Bayless is the man. Why blue corn? It’s a better shell. Crispy but firm. Your tacos won’t fall apart mid-way through eating.

Leftovers from an earlier iteration, before discovering the perfection of blue corn shells. Look at that gorgeous filling!

Another earlier iteration. I was using some mild salsa and tobasco to get toward the heat I wanted. The Frontera Chipotle Salsa produces much better results.

It took a lot of... testing to ensure I had the right Frontera salsa for the job. My favorite remains Chipotle but the other flavors are also delicious and great for chips.

Lots of Tally Counters

Since Tallymander was made a Staff Favorite last month, I’ve noticed that there are more solutions to the tally problem in the App Store than when I began.

There are, of course, many ways to skin a cat. For me, Tallymander does the job best because I built it to my exact desires. Still, while many elements of design are subjective, there are good and bad ways to do things. Let’s look at some of the other approaches to the tally challenge.

Tally Max

A few things jump right out:

Inefficient use of space: The entire width of the iPhone’s screen is available to each tally cell, but the tally title is confined to a much more limited area. The title is the only element that the user can customize beyond the rails of your design — give it some room. Continue Reading…

How to Get Started as an iPhone Developer

See the 2010 updated edition of this post.

Reader Benjamin wrote to me tonight and asked:

I have researched some into iPhone programming as I am obsessed with every application that is available for my own iPhone. The problem is that the amount of books and articles out there about programming for an iPhone is enormous. Do you have any recommendations for a few killer books to read in order to learn the process/language?

What a great question. It’s one I’ve been getting a lot from people I know since my apps went on sale.

Thanks to the popularity of the iPhone and the lure of the App Store’s profit potential, there’s plenty of crap floating around promising to teach you how to program for this new platform. Much of it sucks. Thankfully, there’s some gold to be found for iPhone SDK autodidacts. Let’s check it out. Continue Reading…

Duh: Apple’s Out of the Woods

Last week I got to meet an Apple VP.

Meeting any sort of dignitary from Apple would make my day worth remembering, but this guy was the real deal. He was Apple’s VP of Education, John Couch. John goes far enough back at Apple to have been recruited by a 20-year-old Steve Jobs. This guy worked on the Lisa.

Like I said: the real deal.

Apple was at Full Sail to participate in the announcement of Project Launchbox, a program where students from nearly all our disciplines get MacBook Pro laptops and pro-level software like Logic and Final Cut Studio at very deep discounts. The announcement took place as over one hundred new students — the first of over 4,000 students in the next 12 months — unpacked their new Macs. The students were salivating as they waited to plunge their power buttons for the first time.

Why does our hip but small private college warrant this attention from Apple? It probably helps that Full Sail is the first college to try this on such a massive scale. But it goes deeper than that.

One of the most resonant things John Couch told the assembled mass of students and faculty during the announcement was that education was in Apple’s DNA. And this is absolutely true: so many of today’s most passionate Mac users have memories of the platform — and Apple’s attendant philosophy of user empowerment — that span the decades back toward their childhoods. My own elementary school was loaded to the gills with Apple IIs and eventually with LC 500s. These, plus the help of an SE at home, were the devices that taught me how to be creative.

Continue Reading…