Thursday, June 14, 2007

Agile Stability

(I often write about the "gotchas" with agile, so it's only fair to talk about the victories as well)

The Best Crash , Ever
The other day, our team was conducting our daily play through (a 5 minute demo of the game with the team before our Scrum) and the game crashed in front of us. The team’s immediate response was to stop talking and stare at the 360 dev kit. A few comments came forth such as “wow…what was that?!” or “what the hell is going on?!” I suddenly realized that our company culture had shifted enough to make crashes unacceptable enough to cause surprise and alarm. It was great!
From past experiences, I wouldn’t have expected this level of stability, especially with a project that is a long time away from the shelves. It's been a challenge to get there, but we've had some great visionary programmers focused on building up stablity over the last few years. We've even mocked our attitude towards bugs.

This level of stability can be directly attributed to Scrum and XP practices:

  • Iterations that produce potentially shippable versions of the game.
  • Test Driven Development which surrounds every function of code with unit tests that execute many times during the day.
  • Dedicated build servers for continuous integration and automated testing (unit and functional tests).
  • Stop-the-line culture and tools. When a check-in (code or assets) break the build, every developer knows it because a desktop icon turns red and a jingle plays (such as the Swedish Chef singing his theme song). If your commit was among the last ones, you are pestered about fixing it. We also dedicate programmers, on a revolving basis, to tracking down and fixing crashes. When the game crashes, regardless of the platform it is running on, an email with crash details is sent off and a core dump saved to a network drive. The programmers assigned that day to monitor such crashes jump on the data and try to find the cause as quickly as possible.

Stability is even more important with agile
Iterative development makes stability even more important than it is with phased development. Phased (or waterfall) development doesn’t require an attention to stability or performance until alpha which is the time when various parts are integrated and the game truly runs fully for the first time. On the other hand, iterative development requires input from the working game to adjust the plan going forward from the start. This requires not only stability but performance as well. A game running at 20 FPS is just not as fun as it is running at 30. You can't postpone that.

The Order of Stability in Agile Adoption
Our coach (Mike Cohn) told us up front that there are three stages in adopting agile. This turned out to be absolutely true:
  1. Iteration - The teams adjust to delivering value every iteration.
  2. Engineering - Your engineering practices adjust to support iteration.
  3. Ownership - Teams act like they own their day-to-day work.
Getting used to iteration is straightforward. Changing your engineering processes to support iteration, change and stability takes a bit longer. Ownership requires a shift of culture which can take a very long time (if ever). That is a subject of another entry.


Anonymous said...

We're putting agile practices into place right now... the votes are still out until we finish our release cycle, but it seems to me that agile will uncover issues earlier.

We often suffer with issues that extend development and shorten our quality assurance and testing cycles. This pushes all failure to the end of the cycle.

If agile can get half of the issues out in the open sooner, we'll be much happier.

Clinton Keith said...

That's a good way of stating it. Iterating on potential releases tends to expose every fault sooner rather than later. That transparency drives you to better practices.