Sunday, June 10, 2007

Synchronized sprint focus tests

Reviewing Sprints

Sprints are valuable to show a potentially releasable feature or version of the game to customers. Often who those customers are makes a big difference in the focus of the sprint.

There are three main types of customers who can review a sprint. I'll list them in order of increasing importance:

Studio customers

These are people around the studio, including studio leadership. Often the teams that get the most benefit from studio customers are the functional teams, those teams that are creating tools and underlying technology for the feature or production teams to use.

Publisher customers

These are folks from the publisher. Getting them to participate in the review every sprint can be impossible, but it’s valuable. They haven’t seen the game and so they are less susceptible to the Ugly Child Syndrome. Watch out for the PO with a Plan problem here. Make sure they have a “hands on” person who represents the consumer.

Consumer customers

It’s great to focus test a sprint build. It’s even better to focus the sprint on the focus test. In other words do the following:

  • Let the team know that the sprint build will be focus tested from the start.
  • Build the stories around focus testing
  • Have them participate in the focus testing

Focus testers make the best customers in my opinion.

  • They evaluate just the game
  • The information you gather from them is more honest. They don’t have to worry about hurting the feelings of the dev team especially if you conduct the tests correctly.
  • They are your ultimate customers!
  • Dev teams don’t fear them.

The last point is key. As a “Chief Technology Officer”, I make a lousy customer. I influence salaries, bonuses, promotions, etc in my role. I would love to take that hat off and put the “customer trying to represent the consumer” hat on and discuss the game or feature objectively. However the two roles blur and communication can’t be that fearless.

Note: before the comments pour in on the comment: “I influence salaries, bonuses, promotions, etc in my role” I would like to point out that creating a truly agile incentive and reward system is one of the hardest challenges. It’s not only the practices that need to be changed but some of the deep cultural biases.


When Microsoft was focus testing Midtown Madness in 1999, they were just starting to develop this capability for games. They’d stuff 20 people at a time into a small room with PCs and have them fill out questionnaires. It was a far cry from what they do now, but the value was huge. Conduct some form of focus testing early. It could be as simple as recruiting four gamers at your local GameStop to come over one evening, eat pizza and play the game. You can build up the process and equipment for focus testing over time (neutral questioning, two sided mirror rooms, etc) later (with all the royalties).

No comments: