SimCity mess shows how little faith business should put in beta tests
Beta testers are not the same things as real customers, videogame maker Electronic Arts has belatedly discovered, resulting in a public apology from Lucy Bradshaw, an EA general manager. SimCity is a storied videogame franchise that has been a hit since its debut in 1989, so you’d think EA would have been ready for the horde of customers who mobbed the latest edition of the online-only title. But that wasn’t the case, and many users have been unable to play.
Popularity was only part of the problem for the latest SimCity, notes Bradshaw. Just as important was the fact that real players used the game in ways members of EA’s beta test did not. Details are sketchy, but this much is clear: the people who bought a copy of SimCity were engaging in behaviors that were hugely taxing for EA’s computer servers, forcing the company to double the amount of processing power devoted to the game.
Why didn’t EA see this coming? The simplest answer is that beta tests—and their even more inferior cousin, focus groups, are a terrible way to find out what your customers actually want and how they’ll use your product in the real world.
Malcolm Gladwell wrote about this in his book Blink, noting that the now-legendary Aeron Chair was loathed by early testers. Focus groups, and to a certain extent betas, are a highly unnatural setting in which human—i.e. animal—behavior is distorted by researchers’ expectations and the artificiality of the context in which they’re exposed to a product or idea. They kill innovation and lead to all kinds of inappropriate conclusions—for example, that cold-calling customers shouldn’t work as a sales tactic, and yet it does.
A lot of this comes down to the fact that people are bad at predicting what they’ll really want and, in the end, we don’t really know ourselves all that well. And when you’re talking about the hive mind—all the emergent properties that will come from masses of customers unleashed in an interactive online environment—well, you can’t exactly ask that collective intelligence to tell you in advance what it might like.
Sometimes product designers, like city planners, just need to take control. Electronic Arts
The solutions to this problem fall into a few broad categories.
1. Big data means you no longer have to test with a subset of your customers.
There’s a reason Google has stopped putting the label “beta” on many of its products, even as it unleashes imperfect and unfinished hardware and software on an unsuspecting public. The only way to know what a customer will do is to observe the customer in his or her native environment. In the old days, this was hard; gathering data was expensive and difficult. But the entire point of the age of Big Data is that we no longer have to take a sample of customers and try to extrapolate the behavior of the whole from that.
Big Data is about having all the data, including all the edge cases and the weird stuff that just doesn’t come up with people are trying a product in a closed beta, walled-off focus group or some other sanitized environment.
But wouldn’t throwing open the gates to the latest SimCity simply have crashed it just as thoroughly as EA’s current approach? Not if EA had released the game in a staggered launch. Letting in only as many customers as your servers or distribution capacity can handle is a way to make sure you don’t disappoint the early adopters while also building anticipation in those waiting in line. If EA discovered during a staggered launch that it could only handle a fraction of the users it thought it could, some would have been disappointed by the wait but there wouldn’t have been nearly as big a backlash.
2. Find brilliant designers with the right intuitions.
Nintendo did not invent its company-saving hit, the Wii, by relying on focus groups or beta testers. Instead, a core team of designers built what they thought was a system that the whole family could enjoy, and they were right. Apple is the same—for decades the core of its success was that the company had to satisfy an audience of one: Steve Jobs. As Gladwell has noted before, Madison Avenue advertising companies used to employ a type of cognitive “designer” to evaluate their ads—Freudian psychoanalysts. The subsequent move to “data-driven” approaches in types of advertising where it might be inappropriate, like brand advertising, is in Gladwell’s view a missed opportunity.
3. Try Canada
When Facebook wanted to test its new talk app, it released it in Canada first. Maybe a beta test can work when it involves an entire country, preferably one like your largest market but not so large that it will be unwieldy or unduly influence the opinons of customers in other markets.
Given the size of EA’s launch of SimCity, it seems impossible that the company could have gotten away with option #2. So clearly the company needed to have many more testers in order to avoid a wave of hugely negative reviews of the game. In a way, the company may have succeeded in this strategy: Its subsequent apologies appear to have mollified many users, thus turning its disastrous launch into a kind of accelerated, and imperfect, public test.
More from Quartz
