Tuesday, December 09, 2008

The role of QA on an agile team

"When I was a child we used to suck on pennies...and it was a delight!"

Every year or so I’m invited to speak at one of my son’s schools to speak at one of those parent career day events. It’s always been fun. “Video game developer” is the coolest career imaginable to these kids. I could be standing next to the astronaut-secret agent dad wearing his space suit and they would climb over him to ask me questions. It definitely scores some “dad points” with my boys.

One of the inevitable questions is “how can I be a game tester?”. They believe that the role of tester is eight hours a day of the same experience they have playing games. What could be better than that? Unfortunately I am forced to shoot down this notion. I explain to them that most jobs in the game industry are in very high demand and we get to choose the people with the best grades. Making games requires excellence in everything they are already studying. The math they may not like is what we use to calculate where Mario jumps. The literature class teaches them how to create stories which are essential to games as well. The list goes on. I merely have to point at their work on the walls of their classroom and tie this or that into work that needs to be done making games.

Getting back to the job testing…I describe the long hours with games that crash . I describe how testers often are very frustrated because they have an understanding of what the game is and have very little influence over over how the game is developed. I don’t want to discourage the future generation of testers. I just don’t want to give the impression that there is this wonderful career with a low bar of entry just waiting for them because they are “good at playing games”.

The traditional role of QA

The traditional role of QA on a game project is to pile on a project in its end stage. At this stage, the important decisions about the game have been made and the job of QA is to find and report the defects. Many of the defects are too deep in the design or architecture of the game to be addressed so late. For example, if the “shooting mechanic” just isn’t fun, it may be too late to correct it. Let’s just slap a band-aid on it (e.g. hide the poor aiming behavior with an effect).

As you may guess, the QA role attracts people that are passionate about games. The sheer number of people applying for these role gives the industry the ability to choose among the best. We should be leveraging the value these people bring far more.

The new role

Agile teams can extract more value from testing. Much of this value comes from the team’s goal of delivering a playable game every iteration and the principle that results of the iteration can influence the plan going forward. The QA role shifts to take advantage of this:
  • Rather than sitting in a pool of testers, testers are spread through teams.
  • Stories require QA action for approval before they are considered done.
  • QA’s voice is heard throughout development, not just the end.
  • QA represents the consumer.
The role of QA on an agile team has to change. The traditional approach of “give me the alpha game and I’ll give you bug reports” are gone. The tester’s role on an agile team has to grow. This starts with the tester’s participation in the Sprint planning meeting. When each story is discussed before the tasks are estimated, the tester needs to understand how that story will be approved. This usually takes the form of how the story and the conditions of satisfaction (CoS) are written.

For example, take the following story:
“As a player, when hit the jump button, my character will jump”.
The team would have a discussion about this story and may ask the following questions:
  • Will the character have jump animation? Will it need to be smoothly transition?
  • Will the character be able to jump to a higher level?
  • Can the character jump from a moving platform?
  • Can the character jump while walking or running?
These questions will help the team understand the tasks that will be necessary to implement the story and how to test that the story is actually “done”. For example, the jump story may define the CoS that result from the answer to the questions above:
  • The character will have a simple stand-in animation that will not be blended.
  • The character will be able to jump up to a ledge, or down to a lower level.
  • The character will not be able to realistically jump from a moving platform.
  • The character can jump from any starting motion, but momentum of the player will be ignored. It can only jump straight up.
With these CoS, the tester will be able to verify that the story is complete before the end of the Sprint.

The tester should also catch problems that are not part of specific CoS, but appear as part of development. For example if the test level is missing some physical geometry that allows the player to “fall through the world” the tester should raise the issue as an impediment to testing the game at the daily Scrum.

In addition to testing the game, the tester should help to insure that the team is not impacting other teams through their work. This would include:
  • Performing regression tests on team builds if major changes are about to be committed that might break some obscure part of the game.
  • Testing tools and pipeline changes that a support team might want to release to the content creators.
  • Finding ways to improve the testing process. For example, if tests are frequently encountering a bug associated with bad texture imports, the tester might raise a request to have the automated tester or even the exporters improved to catch the problem earlier.
Much of the tester’s time will be spent verifying the gameplay and offering advice on areas of improvement. This requires an ongoing dialog with members of the team.

Since the tester represents the consumer, they need to keep a “consumer eye” on the game. They should point out any issue that the consumer will notice (such as a bad texture or gameplay progress stopper).

The benefit to the value of what the team adds every Sprint will greatly benefit from having a tester on the team. Customers will get what they ask for and stories will be truly “done”.

The challenge

The tester is challenged in this new role. As mentioned above, the traditional approach doesn’t work on agile teams. The tester is required to communicate with every discipline, understand the tools and stand up and be heard. Not every tester I’ve met can do this. Some I've met are more than ready. QA has traditionally been a “gateway role” for other roles in game development, so it’s an appropriate “test of fire” for people with those skills to grow their career in game development.


Ron said...

Oh, you are so dead on with this entire post it isn't even funny. One of my nephews wants to be a game tester and thinks it is unlimited game playing all day long.

I've done some beta work on the side, and you left out a few things testers deal with: bug reports done via emails vice a tracking system and new releases sent out with no release notes so testers have no idea what was fixed, those were a few of my pet peeves.

I REALLY like your concept of CoS. I'd never heard it before but it makes a lot of sense. It seems easier to understand than, say, 'acceptance tests.' I'll have to try and introduce the idea for our next sprint.

Jacko@TSP said...

This one's a classic - just ran into it again looking for "agile game companies".

A few minor refreshes after having recently come from a very agile team as a QA PM and pinch hitter scrum master.

First, game company QA'ers tend to only see their roles as product testers, quality crusaders or subservient, junior level throw-away workers - instead of credible stewards and evangelists of game AND process quality. In order to "assure" quality, they must have (or take) a seat at the table. Be a credible, diligent, honest partner. Finding defects in units? Institute and start publishing unit test metrics to the team along with bug numbers. Finding too many usability bugs or you notice feature creep impacting the quality of the game? Side with engineers and SM's, stop the line and demand better design requirements. Ask yourself everyday, as QA am I causing any impediments of defects to occur? That's a good place to start.

Another thing I've noticed is that most QA testers all the way to directors have no idea how to truly bring value to the process AND what QA's "product" is. Bugs are not the product, delivery of relevant, meaningful and appropriate coverage, preventing issues, improving process - that's more valuable than jamming thousands of "bugs" in a database for someone else to prioritize. You've heard of XP's "program like hell" - I call that "testing like hell".

If your producers or dev leaders have to question thousands of times if something is a true defect, a suggestion, a "maybe" bug - guaranteed they're not doing something else. Like producing the game.