Embedded QA

I came across the following great article:

http://www.stickyminds.com/sitewide.asp?ObjectId=6851&Function=DETAILBROWSE&ObjectType=COL

This is an entire point that I forgot to talk about: Embedded QA.

As I mentioned at GDC, "thinking Agile" is really hard to adjust to. Our migration of QA is a perfect example of how we struggle with it.

Traditional QA is ramped up during post-Alpha in a traditional waterfall approach to vigorously test the game once all the features are implemented. We've all seen the problems in games where Alpha is the first real chance we have to play the game in some form near it's final state. This often raises the desire to change things even more. This is and example where discovering product value at the end of the project is bad.

Additionally, we underused QA. Good QA people are either frustrated at not having their insights into a game valued ("it's too late to change the game based on your suggestions") or their role is so undervalued in some companies that people are literally hired off the street to fill the role.

We had been doing Scrum for ~ six months before it even occurred to us to reconsider QA. Scrum is about mixing things up. The word is taken from Rugby where the team moves the ball as a group, and doesn't divide up the team into roles as strictly as other sports teams. So why were we separating QA into a separate department that doesn't Scrum with us when our monthly goal was to produce a vertical and complete slice of the game? I can be pretty slow at times, but I eventually "get it".

So we started embedding individuals from QA into the Scrum team. They sat in the same bullpen as the rest of the team. They got their hands a lot more dirty. Some of them could do a bit of coding, others layout and others had some part time associate producer roles, but their core job was to excercise functionality as it came online. It was a huge success.

Our goal with Scrum is to eliminate Alpha/Beta/QA departments, etc (Although we may need a "stabilization Sprint" thrown in to have proper releases). Darkwatch had a rather extended alpha-beta period (having to go through a management buy-out of the studio and shop the game didn't help), so I can't claim we're there yet. Some of our largest remaining problems are:

• Interteam build stability : How to prevent separate teams from harming each other.
• Interteam dependencies : How teams can better support each other.
• Zero bug tolerance.
• Improved automated testing - We implemented this very late in the project.
• Improved handling of bad date in a data driven engine.

More topics to talk about...

Full Product Planning

Another question:

It also doesn't seem terribly useful for estimating the scope of the entire project. Somewhere early on, independent developers have to tell their publishers that they're going to have n levels or stages or whatever, each about so long, and their full-production milestones usually revolve around the delivery of some number of levels. Scrum seems to only be looking a month into the future, so how do we do this part of the process?

The portion of Scrum I focused on in the talk didn't cover long term planning. The main reason for this is that we transitioned to Scrum halfway through the project (a year ago) and didn't get to go through the entire process. However we did manage the product backlog and had experience in using it to manage risk and some planning.

Full production planning (of the product backlog) is a part of Scrum and Agile in general. From what I've read it seems to be able to create a product schedule that you can have far more reliability in than your typical waterfall schedule.

One key point about Agile is its focus on creating product value early. If you look at page 7 of the presentation you'll see the goal of an Agile Project (red) is far steeper than the waterfall curve(blue). This is achieved by placing greater value on working software than a document or schedule.

The long term list of goals and the resultant plan for implementation (including a timeline) can be derived from the product backlog. This backlog is much the same as a document of features except that it is a database of features than can be prioritized and "burned down" by Sprints. One major element of the Scrum I did not talk about was its fractal nature. Look at pages 40-42 of the presentation. This shows a Sprint burndown. You can also "burn down" the product backlog. Since every item in the product backlog has an estimate and you remove some of those items from the backlog every 30 days, then you can produce a chart which shows you how you project is progressing. If your burn-down is slower than your ideal, then you can reduce scope (remove the lowest priority N backlog goals), increase the X axis (get an extension) or increase slope (add more people....the worst choice of all).

In waterfall methods, demonstrated value is delayed a bit more. This can result in core tech being implemented later in the project. This leads to projects entering a "production" phase earlier than they really should. This can lead to delays if the assets need reworking or the release of an unfinished game.

Agile planning (again, speaking from what I've read and not fully practiced) focuses on scheduling what is known and timeboxing/prioritizing what is not. It seems to have better tools for managing risk.

Detailed schedules and docs may give everyone a warm and fuzzy feeling when they are written but like a battle plan, they don't survive first contact with the enemy (in our case, noise).

As we take on planning for our new projects over the next few weeks/months, I'll post feedback here.

Production & Scrum

On the previous entry a poster asked:

Although scrum sounds good for prototyping, and it sounds good once you have a potentially-shippable game that you're polishing, it doesn't seem terribly useful for that phase we call "full production", where (ideally) we're creating a big pile of content using systems that have already been developed. Or am I missing the point?

This is a great point and one that we are trying to address for our next project. I believe that "incremental" development is less useful when you know your requirements and your technology. Take a look at slide # 5 in the presentation. When you are in the situation that you describe above, you are in the "simple" zone of product development. In this case, your process can be more waterfall. You can preplan up front and depend on that plan.

However, this isn't to say that Scrum is not useful in this stage. For example, meeting every day for 15 minutes to discuss progress and impediments is just as useful. Doing a burndown of progress is just as useful. Same goes for having visible progress in the game.

As Ken Schwaber likes to say: "It's about common sense". Scrum encompasses a number of good practices that are as minimal and non-intrusive as possible. If something doesn't make sense to do, then don't do it!

Tracking Individual Estimates

It's funny how you think of the best answer after the fact. One person came up and asked"How can you train people to make the correct estimates if you are not tracking their previous estimates and results".

This is one area that I personally fought over with Scrum. Fourteen months ago, I was in complete agreement with Joel Spolsky on how estimating should be done (which included tracking individual estimates and "actuals"). Now looking back on it, I see that 80% of what he describes is still the core of Scrum. However, the other 20% is still a command-and-control structure.

As I mentioned in the talk, ownership is one of the core benefits of Scrum. I don't think we have the "ownership" thing down yet at HighMoon. This takes time. We have seen the benefit of having tasks broken down and managed by the team. As I showed in the war room picture, the level of detail the team manages is far beyond what any one lead can. As they say, the devil is in the details and if you ignore the details, you'll get killed.

There is one core problem with tracking estimates at an individual level though. This is based on one of the most important lessons I've learned over the last year:

People base their behavior on how they believe they are being measured.

This is evident in how we've seen how small issues with our process has had big impacts on the behavior of people in the teams. This is one of the main reasons I recommend applying all rules of Scrum (at first) and hiring a coach after several months. Good Scrum coaches will look at your practices and can tell you about all those minor things you are doing wrong that have big implications. For example, we thought it was a good idea to record what everyone was saying in every scrum and publish it once a day to the team. The problem that resulted is that people thought they were being measured by the "good news" of what they reported and the scrums stopped being used to raise impediments and just report good news. We followed our coach's reccomendation to stop this saw an improvement (Scrums became more impediment driven). The reports were useful to me and the other directors, but this is one of those areas of Agile that former leads need to adjust to: not doing the command and control thing.

This is the core problem that I see with measuring estimate accuracy. People become concerned about underestimating that they begin to over-estimate to protect themselves. An over-estimated task will rarely be done early (this is called the schoolwork principle or something like that. Basically if you have an extra week than you think to accomplish a task, you will probably slack a bit for the first week).

Here's an example:

1. Task X is created. It should take 8 days for Programmer A to finish if nothing goes wrong.
2. Programmer A doesn't want to deliver late, so he estimates 12 days.
3. Programmer A takes it easy for 4 days. No sense in delivering early.
4. On day 12, an unforeseen risk is realized. It's going to take an extra 2 days to finish Task X.
5. That's OK, because programmer A's manager has a 25% reserve he's added into his estimates he passes up the chain to the master project.
6. Task X is completed in 14 days. Everyone is happy.

So originally, if Programmer A had focused on Task X from the start, then Task X would have been done in 10 days. So our scheduling process has driven up the cost 40%.

Scrum measures product value delivered in the game. This causes a different behavior than measuring estimates. Peer pressure on creating product value is far better than managerial pressure on correct estimation.

Just as with the daily status reports, my command and control side would like to see what everyone's estimation accuracy is, but I don't want to pay the price.

Agile Estimating and Planning

Many of the questions after the presentation were directed at how one plans and estimates the larger project scope using Scrum. This is a major focus for us at High Moon over the next few months as we start new projects using Scrum. We adopted Scrum a year ago and have not gone through a full product dev cycle with it yet.

In antipation of this, I have been doing research and reading a few books. The best one out there is the book to be released by Mike Cohn this summer called "Agile Estimating and Planning" (the link to it is on the AGD website). I've previewed many of the chapters for this book and it seems very pragmatic.

However, there were great questions on how a project should allocate resources over the long-term. I'll comment more on these issues as we plan our next project and begin to work with our new publishers. Each publisher has their own set of expectations.

The bottom line is that we cannot trust the validity of long-term detailed scheduling. Most of us know this and do some form of granular time-boxing anyways. So I don't believe that Agile planning is that different. What is different is to assign a value to each element of your backlog to help adjust the order in which backlog elements are implemented. Also, identifying critical chains in the dev project is important.

More to come....

Initial thoughts on the presentation

On Friday March 11, 2005 I gave my presentation on "Agile Methodology and Scrum in Game Development". Looks like several hundred people showed up. It was a one hour presentation. About 50 minutes of it were taken by my slide presentation (~ 60 slides) followed by questions.

It was a lot of material to present in a single hour. If I do the presentation next year I'd like to expand it to two hours. I'd expand some of the presentation areas (mostly to do with planning) and leave at least 45 minutes for Q&A.

This blog was created to answer any questions that have come up from the presentation and to develop ideas for further developing a future version.

Clint