Sunday, June 17, 2007

No Silver Bullet

A Disturbing Assumption

I occasionally come across comments like this one:

It'll be very interesting to see how High Moon's next game does, because I suspect that the hipness of agile methods in our industry will explode if it's a hit and collapse if it's a failure.

I don’t agree with this prediction either way and we have tried to emphasize this in the presentations:

Agile is not a silver bullet

Some of the key points we stated were:

  • There is no “right way” to make a game.
  • The talent of your team is the main factor in the quality of the game.
  • Agile has benefits:
    • Reducing wasted effort
    • Finding the fun first

I’m not saying:

  • Using agile makes you a hit factory
  • That High Moon has any greater insight than anyone else using agile
    • We’ve just been talking about it for a few years.
  • Waterfall will always fail.

As for High Moon, I’m enjoying myself. Its fun, challenging and we are improving how we make games every day. Will the next game be a hit? I don’t know. It has the strong potential but I’ve given up predicting success solely based on the development side alone. I’ve never predicted it based on the process used to create it!

Let me back that up with a look at a past project I worked on.

A Comparison

One game I worked on was Midnight Club at Angel Studios. It was not a game developed using agile practices which makes for good comparison.

Some facts:

  • Fixed price, fixed schedule development (it was a PS2 launch title).
    • We shipped on time and budget
  • The original scope included six cities
    • We shipped with two
  • The last four months of development was tough.
    • 72 hour weeks for most.
  • The game sold 2.5 million units
    • It was a success for the team and the customer

How did it succeed?

The keys to success were:

  • Talent
    • It was a great team, especially with the technical leadership of Dave Etherton and design leadership of Fred Marcus.
  • Experience
    • The team largely consisted of the Midtown Madness team. Vision wasn’t a problem.
  • Marketing of Rockstar
    • These guys were amazing. Who else would pay to close down Times Square for a night to get box cover shots?
  • Technology
    • Closely tied with talent and experience, but there is benefit to knowing your mature and adoptable technology base from the start. Adopting the PS2 was not easy though.

How would agile have helped Midnight Club?

Agile could have helped mitigate the crunch at the end. Many of us were truly burned out after MC1 and a few good people left. I don’t know if we could have turned out a third city, but dropping that city late in development had a major impact on the flow of the game. Knowing we were going to ship with two cities earlier would have reduced the effort wasted on the third and allowed the designers to adapt the gameplay for two cities.

There are two key points to summarize

  • We succeeded with waterfall.
  • Agile could have helped significantly with crunch and somewhat with gameplay.

Waterfall is not a blank bullet

Waterfall has been rejected largely due to the higher percentage of projects that using it fail (miss budget, quality or schedule goals or are canceled outright). That percentage is not 100% though. Process contributes to but does not determine success.

The point of agile means you shouldn’t accept or reject it blindly

Agile methods use an iterative approach. You are constantly in an “inspect and adapt” cycle. This applies to products, teams, whole companies and even the process itself!

Frameworks such as Scrum or XP suggest practices that support the “inspect and adapt” cycle. What you do with the results of your inspection is up to you. If you find yourself inspecting how teams work together, including everyone in how the operations of the company can be constantly improved and trying to honestly ask yourself if the projects are headed in the right direction on a regular basis, then your shouldn’t care about High Moon’s games!

Scrum is self modifying in nature. If you aren’t achieving the results, then change the process! Just try to preserve the principles though. Too many scrum adopters remove or ignore the principle of change to their detriment.

High Moon’s Next Game

Regardless of the success of failure of our next game, I believe that we are better off using agile to develop it.

If our next game is a hit, does that mean you can do the same by just adopting our practices? No! It means that we had the talent, technology and marketing to make a successful game. If the game fails, we will not blame our process, but ourselves.

The goal is to compete with each other on the basis of talent and not the illusion that we are competing on the basis of who has management that can make their people work longer hours. Toyota writes books about their system and gives competing auto makers tours of their companies and they are still the number one auto maker. Their culture creates success.


Building your studio culture to never be satisfied and to improve your games and how you make games is the ultimate goal. When I read articles like this, I get anxious to help build a culture like that. The irony is that you can't impose this kind of culture. It has to be grown though trust and ownership at every level. These are principles which were used to create agile. Agile can contribute to culture, but it's just a part.

Thursday, June 14, 2007

Agile Stability

(I often write about the "gotchas" with agile, so it's only fair to talk about the victories as well)

The Best Crash , Ever
The other day, our team was conducting our daily play through (a 5 minute demo of the game with the team before our Scrum) and the game crashed in front of us. The team’s immediate response was to stop talking and stare at the 360 dev kit. A few comments came forth such as “wow…what was that?!” or “what the hell is going on?!” I suddenly realized that our company culture had shifted enough to make crashes unacceptable enough to cause surprise and alarm. It was great!
From past experiences, I wouldn’t have expected this level of stability, especially with a project that is a long time away from the shelves. It's been a challenge to get there, but we've had some great visionary programmers focused on building up stablity over the last few years. We've even mocked our attitude towards bugs.

This level of stability can be directly attributed to Scrum and XP practices:

  • Iterations that produce potentially shippable versions of the game.
  • Test Driven Development which surrounds every function of code with unit tests that execute many times during the day.
  • Dedicated build servers for continuous integration and automated testing (unit and functional tests).
  • Stop-the-line culture and tools. When a check-in (code or assets) break the build, every developer knows it because a desktop icon turns red and a jingle plays (such as the Swedish Chef singing his theme song). If your commit was among the last ones, you are pestered about fixing it. We also dedicate programmers, on a revolving basis, to tracking down and fixing crashes. When the game crashes, regardless of the platform it is running on, an email with crash details is sent off and a core dump saved to a network drive. The programmers assigned that day to monitor such crashes jump on the data and try to find the cause as quickly as possible.

Stability is even more important with agile
Iterative development makes stability even more important than it is with phased development. Phased (or waterfall) development doesn’t require an attention to stability or performance until alpha which is the time when various parts are integrated and the game truly runs fully for the first time. On the other hand, iterative development requires input from the working game to adjust the plan going forward from the start. This requires not only stability but performance as well. A game running at 20 FPS is just not as fun as it is running at 30. You can't postpone that.

The Order of Stability in Agile Adoption
Our coach (Mike Cohn) told us up front that there are three stages in adopting agile. This turned out to be absolutely true:
  1. Iteration - The teams adjust to delivering value every iteration.
  2. Engineering - Your engineering practices adjust to support iteration.
  3. Ownership - Teams act like they own their day-to-day work.
Getting used to iteration is straightforward. Changing your engineering processes to support iteration, change and stability takes a bit longer. Ownership requires a shift of culture which can take a very long time (if ever). That is a subject of another entry.

Sunday, June 10, 2007

Synchronized sprint focus tests

Reviewing Sprints

Sprints are valuable to show a potentially releasable feature or version of the game to customers. Often who those customers are makes a big difference in the focus of the sprint.

There are three main types of customers who can review a sprint. I'll list them in order of increasing importance:

Studio customers

These are people around the studio, including studio leadership. Often the teams that get the most benefit from studio customers are the functional teams, those teams that are creating tools and underlying technology for the feature or production teams to use.

Publisher customers

These are folks from the publisher. Getting them to participate in the review every sprint can be impossible, but it’s valuable. They haven’t seen the game and so they are less susceptible to the Ugly Child Syndrome. Watch out for the PO with a Plan problem here. Make sure they have a “hands on” person who represents the consumer.

Consumer customers

It’s great to focus test a sprint build. It’s even better to focus the sprint on the focus test. In other words do the following:

  • Let the team know that the sprint build will be focus tested from the start.
  • Build the stories around focus testing
  • Have them participate in the focus testing

Focus testers make the best customers in my opinion.

  • They evaluate just the game
  • The information you gather from them is more honest. They don’t have to worry about hurting the feelings of the dev team especially if you conduct the tests correctly.
  • They are your ultimate customers!
  • Dev teams don’t fear them.

The last point is key. As a “Chief Technology Officer”, I make a lousy customer. I influence salaries, bonuses, promotions, etc in my role. I would love to take that hat off and put the “customer trying to represent the consumer” hat on and discuss the game or feature objectively. However the two roles blur and communication can’t be that fearless.

Note: before the comments pour in on the comment: “I influence salaries, bonuses, promotions, etc in my role” I would like to point out that creating a truly agile incentive and reward system is one of the hardest challenges. It’s not only the practices that need to be changed but some of the deep cultural biases.


When Microsoft was focus testing Midtown Madness in 1999, they were just starting to develop this capability for games. They’d stuff 20 people at a time into a small room with PCs and have them fill out questionnaires. It was a far cry from what they do now, but the value was huge. Conduct some form of focus testing early. It could be as simple as recruiting four gamers at your local GameStop to come over one evening, eat pizza and play the game. You can build up the process and equipment for focus testing over time (neutral questioning, two sided mirror rooms, etc) later (with all the royalties).

Team Definitions Redux

Definitions that I use for defining the main types of Scrum teams that contribute to a project.

Product Team

The team which owns the game and insures that integration of features, functions and production meet the needs and vision for the game.

Feature Teams

Teams that focus on vertical slices of core gameplay features for a game.

Functional Teams

Teams that focus on horizontal slices of the game (graphics, animation, audio, pipeline) in support of the feature or product teams.

Production Teams

Teams that fill production requirements of the product teams. Very similar in spirit to outsourcing in that iteration is a minimum. Assets are created for known features.

Ugly Child Syndrome

Ever heard the expression that ends with “…only a mother could love”? This applies especially to game development. I’ve worked on a few crappy games in the past (most were mercifully killed before release), but I never thought they were as bad from the team (or even studio) perspective as they really were. Our child was something only we could love.

It’s hard to tell a feature team that their child is ugly, but it is exactly what is needed. The sooner the team knows it, the easier it is to turn it around.

This is the value of having external (publisher or focus test) people who speak their minds play the sprint review build. Customers that see progress on a daily basis start to see beauty where the buyer wont.