Google Tech Talk Video: Mary Poppendieck

Touches a lot of her (and Tom Poppendieck's) topics on Lean. One hour of solid gold.

http://video.google.com/videoplay?docid=-5105910452864283694

Debt and the effects on releases

Our Last Release

Our two main goals of adopting agile is to (a) reduce waste and (b) find the fun first. By focusing on adding scope to the game in prioritized and "completed" increments, we hope to achieve these goals and avoid scrambling at the end to find it.

We develop our games in series of 2-3 week "Sprints". Each Sprint adds "vertical slices" of features that add to the value of the game. The game should be better every 2-3 weeks. We also have "Releases" which occur every 2-3 months (there are about 4 Sprints per Release). Releases are "potentially releasable versions of the game". We think of them as magazine or E3 demos.

As we wrap up another release, I had some observations about how we can do it better. E.g. how we can reduce the waste and improve finding the fun.

With the last 2-month release, we didn't really see much of the value until the final Sprint during which the team elected to work some overtime (the last weekend and some of the nights during the week). Although this was not a death-march in any sense, it's something to keep an eye on towards improving.

So I've been thinking about that.

One of the values of Scrum is its tools for creating transparency in what is going on. One of these tools is the "Release Burndown". This shows how many "Story Points" the team is finishing each Sprint. Story Points are measures of feature difficulty that are relative to one another (and most importantly, not tied to hours). It's been a very good measure of feature velocity when planning future releases.

During the last release, we saw a drop off of Story Point velocity towards the end of the release:
This diagram is used to illustrate what happened (it's not the actual Release Burndown, but simplified to make the point). The blue line shows the story point Burndown. The purple shows the value of the game (fun) from the customers perspective.


Ideally, I'd like to see the value grow at a steady pace. If it's not, then it implies that debt is growing. Some of the work going on is not reflected in the game.

Debt

Debt is unfinished work. We'll refer to it as "parts on the garage floor". Debt can be:

• Polish or tuning work
• assets that are left out or built too soon.
• over-design

If we use polish or tuning as an example, we might see them being ignored in the Sprint builds. The team will take credit for the related story (and it's points) because the functionality is there, but those issues will still need to be addressed. Since it takes time to polish and tune, then the team has a debt associated with that story they thought was complete. This debt has to be paid. Typically this happens at project Alpha, but in our case much of this has to be repaid before the end of the Release.

The reason is that lack of tuning prevents value. How many times have you played a game where an annoying behavior prevented you from enjoying it as much as your could have had it been fixed? You can see the effect of this in the diagram above; As the team shifts from working on stories to working on polish, the story point velocity decreases, but the game value increased at a much increased rate. We customers started enjoying it more and more as those annoying and distracted problems were fixed.

(One comment that comes up is that "all debt is not bad". That's the topic for a future post.)

Why overtime?

So why did we have overtime at the end of the Release? Two reasons:

1. The team elects to do overtime when the product value growth is near vertical and they are running short of time.
2. You do it because you can't ship with critical bugs still in the product.

It's not a great place to be because you are letting the schedule determine the value and QoL. The schedule should determine the scope.

What should happen is:

1. The team completes vertical slices of scope every Sprint.
2. The scope is prioritized.
3. When you near the end of the Release, your Scope velocity determines the Scope cutoff point. The lower value features are left off.

Debt takes the control out of your hands.

Causes? Fixes?

This is the hard part. There are two avenues of how to address this that need to happen:

We customers have to set the expectation for completeness every Sprint. We need to recognize debt and react to it. Recognizing debt in game development is hard.

The team needs to focus on value and not on stories and points. They get fixated on the Sprint Burndown and need to pay more attention to the game.

Fantasic Article on adopting Scrum

One of our programmers found this article. Although this is not a VG developer, the similarities in experience are very close.

GDC AGD Tutorial Announced

The Agile Game Development Tutorial has been announced on the GDC website.

We're still in the process of planning, but the thought is 50% presentations, 50% open discussions (most likely an Open Space).

Article posted

I uploaded copy of an article called "Get in the Game: What others can learn from game developers" on the AGD site. I wrote it a few months back at Mike Cohn's request for "Better Software Magazine". Mike is their technical editor.

Lean Production: first steps

One of our projects is approaching full scale production of levels for the final release version. As mentioned before, we can't be fully Agile here. There are dependencies in levels on functionality than cannot be changed. We cannot "decide to ship or not" every three months unless we have enough gameplay to justify the $50 cost of the game.

At the same time, we don't want to create a huge production plan and blindly commit to it. We have a fixed ship date and we want to know what the quality bar is for the X number of hours of gameplay we want to ship with. Are there any empirical and iterative processes that can help?

Seems so. About a year ago I read: Lean Software Development: An Agile Toolkit for Software Development Managers. The book outlined some of the practices or tools that developers can use that are derived from manufacturing processes (such as 3M or Toyota) called lean production. Lean production is about creating production pipelines that can react to change, defects and improvements very rapidly. Much of lean production derives from Toyota's Production System called Kaizen.

Kaizen directly applies to a level production pipeline for games. Elimination of waste, just in time delivery and a focus on value stream management applies to any production pipeline.

So I was anxious to try this out. Over the past few weeks I've been learning about Value Stream Management. I've found some great materials online as well. However, the best source of information was from Michel Allard, the new "Chief Process Officer" for Vivendi Games. He actually had a great deal of experience with Value Stream Management and came down for a day to help us map out our level production value stream map (VSM).

One rule of thumb for VSMs is that you'll discover that 85% of the work done does not add value! This means that theoretically you could reduce 85% of the cycle time for producing a level. This doesn't necessarily mean you can produce ~six times as many levels, but it points out where you can work on getting the biggest improvements. There are some obvious ones, such as division of labor and reducing the time of one step through an improved tool. There are others, such as approval iterations, that can be improved through better education and communication.

By the end of the day, we had identified steps to achieve a 50% reduction in the amount of cycle time per level. Not bad. Going forward, the benefit for the VSM is to produce a great deal of transparency in each of the dozens of steps and give a great deal of ownership for improvements to the people doing the work. Those are very "Agile" improvements!

Agile failures and how to avoid them

I've been comptemplating a "advice on how to get started" page on the website.

Some very good info on that and the main reasons for failure adopting Agile in this post.

E3 Gone - Part 2

Why did the major 1st parties and 3rd party publishers wanted to pull out of E3. I hear the main reason was that they wanted to "control the message". Take Sony and EA's showing at E3. They were largely considered "lackluster" by the press for their titles and especially what Sony had to show for the PS3.

I don't take those things too seriously. Anyone who recalls how badly Halo showed in 2001 will realize that things still come together at the end for many projects out there (albeit with lots of wasted effort and overtime).

However many people in the press and therefore the consumers do take these things seriously. Sony tried, with their "10% completed" stickers on their kiosks, but people pay attention to what is on the screen and with the Sony representatives vastly outnumbered they couldn't explain in detail the caveats of what was actually running.

Now if all projects had Agile releases and focused on consumer value first it would be a different show. :)

One of the things we learned from the doing Sprint reviews is that the amount of hand-waving and "description of what you are about to see" before we saw the game running is inversely proportional to the consumer value of what was produced during the Sprint.

Maybe this scales. Maybe the change of formats for E3 is an opportunity for marketing to do more hand waving.

Delaying Commitment

There are a number of ongoing areas of confusion about Agile planning and execution. Resolving this confusion is important to avoiding major areas of risk and properly iterating on the game an not a plan.

Some of the questions raised are:

• I know the Wii [or insert big risk here] is a major risk. Why not plan for it and write stories off the plan?
  
• We have a game mechanic that might take a month or two to build. Why not plan it out and write stories off the plan?
  

There are things you plan for; there are others you don’t. How do customers decide that we need to add Wii stories a year before we ship or that we want to take two-week baby steps on a game mechanic?

It’s confusing to the team as well. It’s an inconsistency unless the underlying principles are understood. One of those principles is:

- Decide as late as possible: keep your options open as long as practical but no longer. In other words: “Delay Commitment”.

The customer and the team know that the Wii SKU is a major unknown and it’s of high value. It can’t be fit into a single increment. We have a fixed ship date, which means the scope is the main variable. I suspect that if we delay our commitment to it much longer we may end up with our backs against the wall (schedule). So I would like to learn more about what we’re in for. I would like some goals for the next release to include having the game running on the Wii (a shippable demo) and knowing more about the work that needs to be done to ship the final version.

For the game mechanic, I’m not so concerned about thinking it through and discussing it beyond the iteration time frame. We do that for releases. Where it breaks down is when we commit to the plan from the start. Commitment then takes the form of executing on the components of the plan in parallel to save time when some of those components are dependent on one another. The symptom of what you see when this occurs is what our Chief Design Officer, Chris Ulm, calls “Parts on the Garage Floor”. We’ll see these parts lying around in an iteration review: components of a future working mechanic that add no apparent value to the game. An example of this would be a HUD element that someone thinks the mechanic might need when it gets fully implemented (don’t get me started on the topic of “why HUD elements are admissions of defeat”). It’s an extra part on the floor that we are not sure of. If you put yourself in the position of the consumer looking at it every two weeks and not the developer with a plan in your hear it becomes more apparent. The effort that we put into making that before we know we need it in a working game might have been wasted.

Delaying commitment on a HUD element is good when the mechanic doesn’t need it yet. Delaying commitment on the Wii to the point where the schedule will decide for us is not good.

Functional vs Feature Teams

Before we began using Agile, our project teams were mostly split into teams of programmers, designers and artists. This was great for disciplinary skill sets to be shared. However it increased the overhead of getting things working in the game. When we moved to using Agile, this became far more transparent and needed to be changed. There was initial resistance to this, and this affected our transition to feature teams.

Initially we formed "Functional Teams" that were dominated by one discipline. One example of this would be an AI or Artificial Intelligence" team* that was mostly composed of programmers. After a few months of this, problems emerged. The main problem was that the team was creating AI in a vacuum. The requirements of AI was not being explored in a real game being iterated, but being developed by a programmer dominated team that was focused on delivering an architecture that all AI dependent teams would need. We were seeing architectural progress and not real game value progress after each iteration.

The next step for the teams was to move to "Functional Teams" to "Feature Teams". Feature Teams focus on a vertical slice of game-play (such as a shooting or driving mechanic" and requires a more highly mixed group of developers. The goal is to have a self-contained team that can address every part of development required to iterate on their vertical slice of game.

As we moved from Functional Teams to Feature Teams, the development of core areas of technology became highly distributed. For example, work on AI shifted from one Functional Team creating all of the AI for the project to one or two programmers on each Feature Team. This created problems. The first and foremost was that the technology development fractured into multiple directions. While this fine for early exploration, it was apparent that this would create problems in the future. We were making one game, but with the Features Teams acting independently, we were behaving as through we were creating three separate games. We would eventually need one AI system for the entire game. Also, we saw that feature teams would compete with one another for resources. This did not help create the sense that all three teams were part of one game development project.

The solution was to have teams form around each discipline (in addition to the feature teams). For example, the programmers formed a team with a Scrum Master that met at least once a week (or more often if needed) to discuss the direction of the technical effort of the team. The programming Scrum Master would also meet with the Scrum Master of the other disciplines (art and design) to discuss and come to a consensus on the vision for development for the game. If one feature team needed resources, they were to insure that those resources could be shared for the benefit of the entire team.

* I only use the AI team as an example because it’s clear to relate in a more general way. All teams had this problem.

Agile Backlash

I love Jason Yip's blog. One of his recent posts was a collection of references that had some discussions about "what is wrong with Agile" among other related posts. I loved Feynman's reference to the "Cargo Cult".

AGD Failures

I've heard some stories recently from companies who have "tried Scrum", etc and have given up on it or believe there is minimal value from it. When I discuss the results with individuals from those companies there is a common set of reasons:

1. Scrum was used as a micro-management tool. A command-and-control manager was looking for a way to get more work out of the teams and told them to "use Scrum". The team attempted to implelement it but there was no "Agile champion" or coach around to coach the team and the manager on ownership. Self-organization didn't emerge and it Scrum died on the vine. I refer to this as the "Cargo Cult" approach to adopting Agile :).

2. Too much team ownership emerged. There were no effective customers or product owner and the teams ended up owning their own product. Real product value growth was no where near what was anticipated.

3. The team/company/publisher could not get a handle on planning beyond the iteration. Planning was still done in large documents, long term milestones were still in use, etc.

I hope that we can share some "failure stories" on the mailing list. Not only would the stories help those trying to adopt Agile, but many people have expressed that they'd like to try Agile again in a different environment and don't want to drop thinking about Agile completely.

Back from vacation

Back from two weeks in Colorado. Lot's of great mountain biking out there!

Thanks to everyone who noticed a lack of posts recently ;) I've got plenty to add after I catch up at work.

Clint

Long term planning & Agile?

One of the initial questions raised about Agile development is "how can we plan for a two year project when the plan can change every month?". This is a fair question.

The answer is that you can't plan the exacting details of a project two years out. You can try, which is the major problem facing many game development projects. Typical long-range planning tends to lock down features and adjust the budget, and schedule to match. Agile's approach is the reverse of this. Features are the most flexible "lever" we have in controlling the budget and schedule. Anyone who has been on a team that has had a lot of people added on at the end in an attempt to keep schedule has seen the folly of this. With Agile, we iterate on the features to insure that the feature value we choose to ship with consists of the best features.

Agile projects use "releases" which are version of the game that are potentially "release-able" every three to six months. Think of a release as a version of the game you would publish on a game magazine's disc. It's debugged, optimized and passes some of the more significant TCR/TRC tests.

Releases are superior to milestones because they are true measures of the game's value. They can be focused tested and shown to larger audiences at E3. Everyone that has developed a demo for E3 knows the value of having the team drive towards the goal of getting a real product on the floor for that show. You discover the reality of what you thought was valuable and what was not building that demo.

Great Dr Dobbs Article

I read this great article on selling XP to traditional (risk cautious) organizations. It addresses some of the reality vs. rhetoric out there. What really caught my eye was the cost of change curve on page 2. I've always thought of the XP CoC over the entire project cycle vs. the waterfall CoC curve. What he shows here simplifies that to one curve with the x-axis not the project time frame, but the "length of feedback cycle". He writes: "Comparing the cost-of-change "curves." The agile and traditional cost-of-change curves are identical. The difference is that agilists follow techniques that keep us at the low-cost end of the curve."

I like that it applies the CoC curve to all of Agile, not just XP.

High Moon "Insider's view" of Scrum

A tounge-in-cheek video on our use of Scrum (hat's off to "The Office")

http://www.highmoonstudios.com/

The video will be there until ~5/8.

Enjoy!

Great article on game dev woes

This great article by Jason Della Rocca, the tireless Executive Director of the International Game Developers Association just came out. It had a link to AGD which explained the big spike of page views on the same day it was published! We all pay attention to what Jason has to say.

SDSIC Presentation

I'll be making a presentation at the San Diego Software Industry Council on April 27th on Agile as applied to game development. I'm combining slides from my GDC presentation with slides from my Scrum Gathering presentation.

The presentation is interesting to me because previous presentations have been to:
- Game developers about Agile.
- Agile experts and coaches about game development using Agile.

and now to an audience unfamiliar with both.

Ken Schwaber Post on Yahoo Scrum Development list

Great stuff...
----
Scrum is really simple, barely a process, more a framework. The hard work in using Scrum is fixing the things that it exposes, actually inspecting the things that it makes transparent and adapting to optimize the results and the organization that produces the results.

Scrum is not the organization that produces the results, or the amalgam of procedures, tools, automation, and standards that are implemented as a result of the inspection, as part of the adaptation. Scrum is the very simple mechanism that helps an organization be more effective in accomplishing its goals.

I've been following the threads about type N, A, B, C and advanced Scrum. Although these may represent the engineering, personnel, and product management practices that an organization adopts as a result of Scrum's inspect and adapt, they aren't Scrum. I think we are mistaking the consequences of Scrum with Scrum itself.

What may be most destructive about these supposed extensions is that they will divert people from the real work of Scrum ... seeing what is going on in their organization and going through the change process to become effective. And learning how to continually inspect and adapt to keep their organization's practices optimal. Instead, people may think that all of these things that use the Scrum name are advances in Scrum, templates that they can mimic and then, amazingly, they are advanced development organizations, also.

We are running the danger of any small process. People want to make it bigger. Well, Scrum isn't bigger. Each organization's total ability to build complex products is certainly bigger, and hopefully continually evolving, but it isn't Scrum.

Keep Scrum Simple.
Ken

Hours vs. Story Points

In my talk, I briefly described User Stories and Story Points. I could have easily spoken for an hour on User Stories, but I could only spend a few minutes. To highlight, User Stories are:

• Feature descriptions from anyone on the team or any customer.
• Use a simple template to insure correct communication.
• Have "Conditions of Satisfaction" which are things that can be seen or tested in the review.
• Are estimated.
  
• Are not dependent on other stories.

One of the key values of User Stories is that they can be estimated. Estimations are made using "Story Points". Story Points are a relative measure of feature difficulty/complexity that are useful for planning. If one feature takes twice as much effort as another, it will have twice the story points. As it turns out, Story Points are a more accurate measure of project velocity and release schedule than using hours and days.

"Hours and days" attempt to not only estimate the effort of a feature, but also the velocity of the people working on it. With Agile Planning, we separate those two measures. We estimate the effort of the feature in isolation and emperically measure the velocity of a team that is doing the work. There are many benefits of doing this. They are separate numbers that come from separate factors!

One major benefit of estimating stories is using the "poker game". The poker game is where the team and customers get together and discuss the stories. Each story is discussed and then each person at the meeting raises a card that shows how many story points they feel should be assigned to the story. The first vote usually shows a wide variation of points. We then discuss the estimates (usually the high & low voting people talk first). This is very beneficial because the unknowns and assumptions are revealed. People take very different views of a feature and this leads to wide variations of expectancies when the feature is estimated. All of this comes out in the discussion of the story being estimated and some of the points make it in as "conditions of satisfaction" for the story.

This process is repeated until the room is pretty close on what the points should be and then we move on.

To estimate an entire release (4 months) of stories took a dozen people an entire day (we had to go offsite so we wouldn't be interrupted). We also have "lookahead" planning sessions where the entire team gets together about once a month to refine the stories (disaggregate larger stories into smaller ones) and estimate those stories.

The end benefit is that teams get a consistent story burndown velocity that allows them to see how well they are tracking their release goals (measured once every 2-week sprint). This is the velocity of real feature development that is far more valuable than estimated hours.

GDC 2006

Thanks to everyone that came to the 2006 session! Lots of great questions.

The slides are posted on the main page of the website.

I'll have some follow-up posts coming up soon. So much detail is left out of a one hour presentation. I have a lot to add.

Mistakes Implementing Scrum

An honest blog entry on what can go wrong when implementing Scrum. I imagine this type of problem is the most prevalent (shifting from command-and-control was a major hurdle for us).

I suggest that when adopting Scrum, you do all of it and follow the rules. Scrum is meant to be modified (especially for game development), but if you modify it from the start you are making changes to a system you might not understand and are probably embedding some organizational flaws into it. This was raised by Ken Schwaber to our last round of CSM grads.

A Day In The Life Using XP

Noel (R&D lead architect at High Moon) has posted the GD Mag article he wrote about the day in the life of programming using XP. It gives a lot of insight into one of the Agile processes we use at HMS.

http://www.gamesfromwithin.com/articles/0602/000104.html

GDC 2006 Presentation

The schedule for GDC 2006 sessions are up. My presentation this year:
Agile Methodology in Game Development: Year 3
is Thursday (March 23, 2006) 2:30pm — 3:30pm

Mention this blog and you get free admission to Waterfall 2006!

I've read a great book Beyond Bullet Points and have modified the presentation based on many of the tips in it.

See you there!

Clint

New conference

Waterfall2006.

Enjoy.

Scrum Master Tip

With all recent CSM grads and visit from Mike Cohn, we're making a lot of tweaks to the process. One great tip came from Mike (based on a question from a new Scrum Master):

People on a team will get into the habit of answering the three Daily Scrum questions by addressing the Scrum Master directly. This reinforces the "Scrum Master is the Lead" mind-set, which is not what you want. Each team member is addressing the team as a whole, making a commitment to them.

The tip is for the SM to avoid eye contact. Look at the ground, at the task board, etc. This forces the speaker to address others in the team.

Works like a charm!

Certified Scrum Master Training

We sent nine people to CSM training last week. I can't recommend it enough. They see the light. The one practice I'd suggest is to get them together in a room and talk about what they learned and reflect on how the company is using Scrum. There's not "one way" to do Scrum. You are encouraged to change the practice as you use it to fit your environment. It is useful to "follow the rules" at the beginning to learn the reasons for Agile let team ownership soak into the culture.

However new CSM grads can be a bit fanatical. Talk it through before they hit the team!

Retrospectives

We've started doing retrospectives (I'm tired of hearing the gasps when I tell experienced CSMs that we weren't doing them).

Whare are retrospectives? It's a team meeting that takes place after a Sprint review. The team gets together and answers the following questions:

1. What should we start doing?
2. What should we stop doing
3. What should we continue doing?

The team reflects on their practices and things which affect their team. They pick a number of things to start and stop to work on over the next sprint. These things show up as "continue" items (along with other good practices) at the next retrospective.

This meeting should be time-boxed to 30 minutes. Since we were doing our first (with our consultant Mike Cohn in attendance) we took more time. A LOT of interesting stuff came out of it!

Evolution of Scrum Teams

In adopting Scrum, we've gone through a number of evolutions of the makeup and backlog of Scrum teams:

- Functional Teams
Teams that are mostly programmers, artists or designers. This rose due to the early adoption of Scrum by the programmers. Art and design participated, but still worked from a schedule (especially in the last year of Darkwatch).

- Feature Teams
Teams that focused on a feature (such as AI, animation, levels, etc). These teams didn't change much, but started to look like mini-teams that were more self-contained. The problem with them is that you get uneven production of features and delayed integration (e.g. AI in levels with good animation).

- Subgame Teams
I'm tempted to call these "demo teams", but many times "demo" implies shortcuts that you wouldn't ship. These teams produce completed portions of the game in releases that you could use for magazine demos. Usually each subgame has 2-4 teams working on it. The goal is to have a version of the subgame every 1-3 sprints (with separate teams showing their work at reviews otherwise).

This evolution leads to teams that show true product value more frequently.

Scrum FAQ at Conchango

Conchango has been doing some great work with Ken Schwaber on creating online resources for Scrum. They've created a great website to introduce Scrum (with video clips by Ken). They've just released a great Scrum FAQ. This is a must see site for developers new to Scrum.

High Moon Acquired by VUG!

Great news! Why?

- VUG has been very enthusiastic about Scrum. Other VUG developers have started using it.
- Their product vision is great. Can't talk about the work we are doing with them, but it's very exciting.

We have a very talented group of people at HMS, super culture and a great facility to grow into and I'd like to think that continuous high value delivered every Sprint was a selling point as well.