Sunday, December 17, 2006

Debt and the effects on releases

Our Last Release

Our two main goals of adopting agile is to (a) reduce waste and (b) find the fun first. By focusing on adding scope to the game in prioritized and "completed" increments, we hope to achieve these goals and avoid scrambling at the end to find it.

We develop our games in series of 2-3 week "Sprints". Each Sprint adds "vertical slices" of features that add to the value of the game. The game should be better every 2-3 weeks. We also have "Releases" which occur every 2-3 months (there are about 4 Sprints per Release). Releases are "potentially releasable versions of the game". We think of them as magazine or E3 demos.

As we wrap up another release, I had some observations about how we can do it better. E.g. how we can reduce the waste and improve finding the fun.

With the last 2-month release, we didn't really see much of the value until the final Sprint during which the team elected to work some overtime (the last weekend and some of the nights during the week). Although this was not a death-march in any sense, it's something to keep an eye on towards improving.

So I've been thinking about that.

One of the values of Scrum is its tools for creating transparency in what is going on. One of these tools is the "Release Burndown". This shows how many "Story Points" the team is finishing each Sprint. Story Points are measures of feature difficulty that are relative to one another (and most importantly, not tied to hours). It's been a very good measure of feature velocity when planning future releases.

During the last release, we saw a drop off of Story Point velocity towards the end of the release:
This diagram is used to illustrate what happened (it's not the actual Release Burndown, but simplified to make the point). The blue line shows the story point Burndown. The purple shows the value of the game (fun) from the customers perspective.

Ideally, I'd like to see the value grow at a steady pace. If it's not, then it implies that debt is growing. Some of the work going on is not reflected in the game.


Debt is unfinished work. We'll refer to it as "parts on the garage floor". Debt can be:
  • Polish or tuning work
  • assets that are left out or built too soon.
  • over-design
If we use polish or tuning as an example, we might see them being ignored in the Sprint builds. The team will take credit for the related story (and it's points) because the functionality is there, but those issues will still need to be addressed. Since it takes time to polish and tune, then the team has a debt associated with that story they thought was complete. This debt has to be paid. Typically this happens at project Alpha, but in our case much of this has to be repaid before the end of the Release.

The reason is that lack of tuning prevents value. How many times have you played a game where an annoying behavior prevented you from enjoying it as much as your could have had it been fixed? You can see the effect of this in the diagram above; As the team shifts from working on stories to working on polish, the story point velocity decreases, but the game value increased at a much increased rate. We customers started enjoying it more and more as those annoying and distracted problems were fixed.

(One comment that comes up is that "all debt is not bad". That's the topic for a future post.)

Why overtime?

So why did we have overtime at the end of the Release? Two reasons:
  1. The team elects to do overtime when the product value growth is near vertical and they are running short of time.
  2. You do it because you can't ship with critical bugs still in the product.
It's not a great place to be because you are letting the schedule determine the value and QoL. The schedule should determine the scope.

What should happen is:
  1. The team completes vertical slices of scope every Sprint.
  2. The scope is prioritized.
  3. When you near the end of the Release, your Scope velocity determines the Scope cutoff point. The lower value features are left off.
Debt takes the control out of your hands.

Causes? Fixes?

This is the hard part. There are two avenues of how to address this that need to happen:

We customers have to set the expectation for completeness every Sprint. We need to recognize debt and react to it. Recognizing debt in game development is hard.

The team needs to focus on value and not on stories and points. They get fixated on the Sprint Burndown and need to pay more attention to the game.


Anonymous said...

I'm by no means a scrum expert, nor even the user, but doesn't scrum rules say that you have to deliver fully tested "potential releases" in the end of every sprint? Also, you shouldn't mark stories as compleate if they still have bugs, and do not deliver the intended value. This way you will avoid getting in debt with product's value.

Clinton Keith said...

You're right, but the bugs that we see in game development aren't the traditional bugs that are simple binary ones that can easily be identified. Many are of a qualitiative or subjective nature. It gets back to what is "fun" as well.

Clinton Keith said...

I reworded it slightly to reflect that these were "tuning and polish" issues rather than bugs. We call them all bugs but I realize this convention isn't shared. Thanks.

Anonymous said...

Yeah, i've been thinking about polishing part and how it would integrate with game development using scrum a lot. I think there are two ways of handling it.

First, as i've already mentioned, is to leave the story as unfinished if it's not compleate (polished enough) yet. Since teams are crossfunctional, game designers in a team could have a final word on such stories, where word "finished" is a bit vague and involves "fun" factor. Or lead artist could have a final word if assets are of required quality, etc.

Secondly, since scrum proccess is iterative and designed for fast feedback, you can use that feedback at the end of the sprint to add new stories into product backlog for polishing the existing features. If they have enough value, they could be pulled in the successive sprints.

Clinton Keith said...

This is where "by-the-book" breaks down.

The first solution doesn't work with cross-functional teams as well as you might think.

The second solution doesn't address the problem of carrying debt. It's close to what we do now.

The current solution is to have discipline leads on each team that show versions of the game to the customers twice a week and get feedback. This is very command-and-control "by-the-book". The question to me is how to build this feedback cycle into a self-organizing team without impacting commitment.

Jamie said...

Thanks for this -
It's possible constant velocity towards release is simply not of the reasons Schwaber's book appeals to me so much is he simply recognized that scheduling software in MS Project is not doable, so maybe we need to take a page from him and accept that we're going to see this curve. (We're seeing a similar curve on our small team - velocity started at an exciting 2 and is now at 1...who knows where it will be in a few months.)
And if that's the case, maybe some higher math could give us a better guess at how many story points we're going to have done by release...

Clinton Keith said...

Agreed. I think we can get a pretty steady "story-point per release" measure that should stay near constant (or steadily improve) on a release to release basis, but not per sprint.

There's been some good discussion over in the yahoo scrumdevelopment list along similar lines. Mike Cohn makes some good points about not relying too much on "story points per sprint". Stories that are small can be far more accurately estimated than the larger stories. Plus it seems the inaccuracy is on the underestimation side.

-tom! said...

This is an interesting and thought-provoking article. I wonder if some (or much?) of the debt due to team members claiming that a bit of the product is done when it really isn't done-done integrated including tool path? E.g. sure, the code is technically complete, but there doesn't exist any means by which the functionality can be exploited except in very limited (and perhaps hard-coded or on the developer's machine only) ways?

My experience with Scrum is across a wide variety of environments and teams, and one of the things that as a manager I've been mindful to ask the person as a task was being moved to the completed column, "ok, so how does a content creator see that in the game right now?" Do you have to deal with this issue at High Moon, now that you've been doing Agile Things for some time?