Thursday, December 22, 2005
Wednesday, December 07, 2005
Tuesday, November 22, 2005
High Moon was noted for its adoption of an innovative product development
methodology known as Scrum. Named after the tight line of scrimmage
formation in rugby, the Scrum method is a process derived from Agile
Methodologies established by software and high-tech R&D companies. Scrum
promotes development through interdisciplinary, non-hierarchal teams who
work in short cycles around demonstrable goals. It addresses the
disciplinary divide among artists, designers and programmers as well as
long development cycles that hinder product evaluation, making it well
suited to game development. Scrum also provides a framework for enhanced
productivity during the course of a standard workweek, giving each team the
ownership to establish its own objectives and a strategy to reach them for
each project cycle.
High Moon Studios, developers of Old West vampire-themed FPS Darkwatch, has received a 2005 Workplace Excellence Award from the San Diego Society for Human Resource Management. The company received a Crystal Award for "Employer of Choice," a category which had 140 other candidates vying for the prize. "We have talented people on staff, and have become accustomed to having their achievements recognized by awards. Yet this award recognizes their satisfaction with being part of High Moon, which makes it an exceptional honor for the entire company," said High Moon CEO John Rowe.
Saturday, November 19, 2005
I gave my keynote presentation on Wendesday morning. The game industry has lots of fun stories to tell that apply to traditional waterfall management and we have some fun stories to tell about our Scrum experiences as well...it was a good experience to share those.
The gathering was very informative and enjoyable. The open circle discussion format was effective for creating discussion topics.
Two impressions were reinforced at the gathering that I'd like to note: 1) we share the same challenges and successes that other, very different, companies using Scrum experience and that 2) Scrum is perfect for Video Game Development.
More notes to come. Have a look at the wiki.
Thursday, September 29, 2005
I've been invited to talk about Scrum in gamedev at the "Fourth Scrum Gathering".
Saturday, July 23, 2005
We're several months into two new project with major significant deliverables finished and very little overtime experienced. The main difference I see is with our good velocity of progress. This can be attributed to the intensity of the 2-week Sprints early on. This intensity drives real deliverables and real daily (8 hour) intensity. If we can focus on product value early, reduce the risk early and not save up our debugging and tuning for the end, we can keep this high-intensity, low overtime pace up for the entire project.
The goals of a Sprint create an increase in intensity every two weeks for the review. Waterfall intensity will peak every milestone deliverable, but since you are focused on documentation up front (which is typically lower intensity) and integration/debugging at the end (which is higher intensity coupled with the project deadline), then you get a trend like the red line. When intensity rises beyond a certain level, someone in the chain of command is going to push the "mandatory overtime" button.
This is not to say that Scrum avoids overtime. One team did put in a weekend day at the end of their Sprint recently. This was a decision by the team to meet their commitment.
Also I wouldn't say that we have the XP engineering practices down pat yet. We're building a backlog of bugs that will bite us at the end unless we continue to improve unit and functional testing. This is a big priority now.
Sunday, June 19, 2005
- XP/TDD: Our R&D group has been using XP & TDD for a few months and other teams are following. The consistent feedback I get is that the initial fear of pair-programming and TDD quickly fade. In fact once you are "infected" with the bug to do TDD, you won't go back. I see now the necessity of doing pair-programming; the low-level yet persistent discipline to doing TDD the right way is reinforced when your partner is watching you ;)
Velocity for each programmer doing PP/TDD is about 60% of normal, but the code quality and stability pays for that 40% & more.
- User Stories: Our consultant (Mike Cohn) gave a class on User Stories and it was very useful. By following some of the simple templates/rules for user stories, teams and customers can greatly narrow the assumptions made in Sprint planning.
- Two week Sprints in early product development: We switched to two week Sprints based on the rule of thumb to make Sprints only as long as your customers can keep their hands off the team. I swear the velocity is almost the same as month long Sprints...so many things have changed that it's hard to know the reason for the increased velocity.
- Long term planning: this was a workshop following the User Story class. We were introduced to a great method of estimating user stories in terms of "story points" which is a relative rating of the difficulty of stories or "Epics" (larger stories that cannot fit in a Sprint. The method involves groups of pigs and chickens raising story point cards (the larger the number, the harder the story) and then discussing the assumptions to work out the differences. When your product backlog is planned out (Mike suggests in 6 month releases) then you can burn-down and measure story point velocity on a Sprint to Sprint basis. Very useful and Agile method.
Saturday, May 07, 2005
Although the dividing line isn't hard cut, a game development project can be divided into two phases with very different characteristics. These phases are pre-production and production. Much discussion has recently occurred on some Agile lists about the unique requirements of game development. The question is "How Agile can we be during production?".
Mark Cerny's Method presentations and articles describe a great method of an incremental and iterative pre-production phase followed by a more classical (higly planned) production phase.
Cerny's Method is an answer to the problem that "you can't be completely Agile in production". E.g. you should define "character height" at the start of production, but you definitely need to know the best character height by the time you enter production and start making levels that depend on it. You shouldn't iterate on character height and all the other values that levels are dependent on after you start making the levels.
Cerny's Method also promotes heavier documentation and planning for the production phase. As you can imagine, such an approach for production doesn't sit well with some of the Agile methodologists out there. I don't see that there is a conflict. Agile is an answer to uncertainty in requirements, tech and resources.
Preproduction is the Agile phase that tries to remove uncertainty from technology and create agreement in requirements so that we can enter production with good game-play. The idea that this chart demonstrates is that the level of chaos guides the process chosen. High complexity&uncertainty drive the need for Agile methods. However pre-production removes technical uncertainty and creates requirement agreement. After pre-production you can choose a different method if it's better.
If you still have technical uncertainty and lack of requirements agreement, you have no reason to be in production (other than trying to get paid for a publisher milestone ;)
The "Simple" area of the graph is actually the area where more detailed planning works well. Write the big script and level design documents. Create the complex Gantt charts. Schedule the details. Some Agile practices still work well. Pair work, daily meetings, burndown charts are all still valuable.
This approach potentially answers some of the uncertainty of doing long-term planning. I'll have a number of future posts addressing this.
Sunday, April 03, 2005
Agile methods define the roles of the customer and the developer very clearly. The developers don't work on anything that is not clearly defined by the customer. These definitions include specific user stories and acceptance criteria for determining when a requested feature is complete.
I saw the following diagram in Agile Project Management that clearly shows the relationships between the customers and the developers.
There are major benefits for both groups. The customers get clear control over the priority and definition of the features which determine product value. The developers get control over the mechanics of implementing the feature, including the schedule.
The developers don't get handed a 300 page document and a time-drame and budget to complete the game defined by the document. Every iteration they allow the customers to decide how they want to spend the iteration budget. If they request too much for that iteration, the developer let's them know in planning that they need to reduce the scope of their request. Once the iteration plan is made, the developers go off on their own to implement the stories without being disturbed with changed goals, etc.
Customers in the game development world are not only the publishers, but the designers and artists that will use systems developed by tech. Additionally design, art and programmers assume the developer role to polish and tune features (with programmers) to be accepted at the end of an iteration.
One interesting thing has come up with our R&D efforts. Since R&D will produce core tech that will be used by the production teams, the programmers that will use the product of R&D are the customers of that product. Having "programmer customers" and "programmer developers" trying to hammer out things such as intermediate file formats and APIs can be challenging. This is where having the CTO as the "Product Owner" of R&D comes in.
Saturday, April 02, 2005
This week we’ve launched our first XP iteration. While we’ve been using parts of XP for a while, this has forced us to consider some of the differences between XP and Scrum iterations/Sprints.
First, the rollout of XP has been pretty smooth. One concern was that the introduction of pair-programming and test driven development would meet a lot of resistance and suffer in effectiveness. This hasn’t been the case. In fact, it’s been embraced by people transitioning over to XP.
Part of this can be attributed to lessons learned over the last year. Creating our first generation data-driven engine for a finished game in 25 months had left a lot of loose ends. One of the biggest pitfalls of such architectures is the ability for bad data to cause problems unless you are really careful. We didn’t have the resources or the practices to be as careful as we needed to be. Pain is a great teacher and it’s clear that we need to embrace better engineering practices to avoid this in the future. With the successful application of one Agile method (Scrum), we’re more than willing to try out XP as a solution.
XP books say that you can roll out XP practices one or two at a time, and we’ve done this. Pair programming and unit testing were the first practices introduced. Now we’re introducing proper iterations (as we grow the first teams that need it). Clearly we have work to do in writing user stories and defining acceptance tests.
The good thing is that you can come out of an XP iteration planning “game” with a bunch of tasks cards and start doing Scrum as usual. There is nothing inconsistent with either practice other than the 2-week Iteration versus the 4-week Sprint. However both methods allow you to adjust your iteration time range completely.
Friday, March 25, 2005
As I mentioned at GDC, "thinking Agile" is really hard to adjust to. Our migration of QA is a perfect example of how we struggle with it.
Additionally, we underused QA. Good QA people are either frustrated at not having their insights into a game valued ("it's too late to change the game based on your suggestions") or their role is so undervalued in some companies that people are literally hired off the street to fill the role.
We had been doing Scrum for ~ six months before it even occurred to us to reconsider QA. Scrum is about mixing things up. The word is taken from Rugby where the team moves the ball as a group, and doesn't divide up the team into roles as strictly as other sports teams. So why were we separating QA into a separate department that doesn't Scrum with us when our monthly goal was to produce a vertical and complete slice of the game? I can be pretty slow at times, but I eventually "get it".
So we started embedding individuals from QA into the Scrum team. They sat in the same bullpen as the rest of the team. They got their hands a lot more dirty. Some of them could do a bit of coding, others layout and others had some part time associate producer roles, but their core job was to excercise functionality as it came online. It was a huge success.
- Interteam build stability : How to prevent separate teams from harming each other.
- Interteam dependencies : How teams can better support each other.
- Zero bug tolerance.
- Improved automated testing - We implemented this very late in the project.
- Improved handling of bad date in a data driven engine.
Tuesday, March 15, 2005
It also doesn't seem terribly useful for estimating the scope of the entire project. Somewhere early on, independent developers have to tell their publishers that they're going to have n levels or stages or whatever, each about so long, and their full-production milestones usually revolve around the delivery of some number of levels. Scrum seems to only be looking a month into the future, so how do we do this part of the process?
The portion of Scrum I focused on in the talk didn't cover long term planning. The main reason for this is that we transitioned to Scrum halfway through the project (a year ago) and didn't get to go through the entire process. However we did manage the product backlog and had experience in using it to manage risk and some planning.
Full production planning (of the product backlog) is a part of Scrum and Agile in general. From what I've read it seems to be able to create a product schedule that you can have far more reliability in than your typical waterfall schedule.
One key point about Agile is its focus on creating product value early. If you look at page 7 of the presentation you'll see the goal of an Agile Project (red) is far steeper than the waterfall curve(blue). This is achieved by placing greater value on working software than a document or schedule.
The long term list of goals and the resultant plan for implementation (including a timeline) can be derived from the product backlog. This backlog is much the same as a document of features except that it is a database of features than can be prioritized and "burned down" by Sprints. One major element of the Scrum I did not talk about was its fractal nature. Look at pages 40-42 of the presentation. This shows a Sprint burndown. You can also "burn down" the product backlog. Since every item in the product backlog has an estimate and you remove some of those items from the backlog every 30 days, then you can produce a chart which shows you how you project is progressing. If your burn-down is slower than your ideal, then you can reduce scope (remove the lowest priority N backlog goals), increase the X axis (get an extension) or increase slope (add more people....the worst choice of all).
In waterfall methods, demonstrated value is delayed a bit more. This can result in core tech being implemented later in the project. This leads to projects entering a "production" phase earlier than they really should. This can lead to delays if the assets need reworking or the release of an unfinished game.
Agile planning (again, speaking from what I've read and not fully practiced) focuses on scheduling what is known and timeboxing/prioritizing what is not. It seems to have better tools for managing risk.
Detailed schedules and docs may give everyone a warm and fuzzy feeling when they are written but like a battle plan, they don't survive first contact with the enemy (in our case, noise).
As we take on planning for our new projects over the next few weeks/months, I'll post feedback here.
Although scrum sounds good for prototyping, and it sounds good once you have a potentially-shippable game that you're polishing, it doesn't seem terribly useful for that phase we call "full production", where (ideally) we're creating a big pile of content using systems that have already been developed. Or am I missing the point?
This is a great point and one that we are trying to address for our next project. I believe that "incremental" development is less useful when you know your requirements and your technology. Take a look at slide # 5 in the presentation. When you are in the situation that you describe above, you are in the "simple" zone of product development. In this case, your process can be more waterfall. You can preplan up front and depend on that plan.
However, this isn't to say that Scrum is not useful in this stage. For example, meeting every day for 15 minutes to discuss progress and impediments is just as useful. Doing a burndown of progress is just as useful. Same goes for having visible progress in the game.
As Ken Schwaber likes to say: "It's about common sense". Scrum encompasses a number of good practices that are as minimal and non-intrusive as possible. If something doesn't make sense to do, then don't do it!
Sunday, March 13, 2005
This is one area that I personally fought over with Scrum. Fourteen months ago, I was in complete agreement with Joel Spolsky on how estimating should be done (which included tracking individual estimates and "actuals"). Now looking back on it, I see that 80% of what he describes is still the core of Scrum. However, the other 20% is still a command-and-control structure.
As I mentioned in the talk, ownership is one of the core benefits of Scrum. I don't think we have the "ownership" thing down yet at HighMoon. This takes time. We have seen the benefit of having tasks broken down and managed by the team. As I showed in the war room picture, the level of detail the team manages is far beyond what any one lead can. As they say, the devil is in the details and if you ignore the details, you'll get killed.
There is one core problem with tracking estimates at an individual level though. This is based on one of the most important lessons I've learned over the last year:
People base their behavior on how they believe they are being measured.
This is evident in how we've seen how small issues with our process has had big impacts on the behavior of people in the teams. This is one of the main reasons I recommend applying all rules of Scrum (at first) and hiring a coach after several months. Good Scrum coaches will look at your practices and can tell you about all those minor things you are doing wrong that have big implications. For example, we thought it was a good idea to record what everyone was saying in every scrum and publish it once a day to the team. The problem that resulted is that people thought they were being measured by the "good news" of what they reported and the scrums stopped being used to raise impediments and just report good news. We followed our coach's reccomendation to stop this saw an improvement (Scrums became more impediment driven). The reports were useful to me and the other directors, but this is one of those areas of Agile that former leads need to adjust to: not doing the command and control thing.
This is the core problem that I see with measuring estimate accuracy. People become concerned about underestimating that they begin to over-estimate to protect themselves. An over-estimated task will rarely be done early (this is called the schoolwork principle or something like that. Basically if you have an extra week than you think to accomplish a task, you will probably slack a bit for the first week).
Here's an example:
- Task X is created. It should take 8 days for Programmer A to finish if nothing goes wrong.
- Programmer A doesn't want to deliver late, so he estimates 12 days.
- Programmer A takes it easy for 4 days. No sense in delivering early.
- On day 12, an unforeseen risk is realized. It's going to take an extra 2 days to finish Task X.
- That's OK, because programmer A's manager has a 25% reserve he's added into his estimates he passes up the chain to the master project.
- Task X is completed in 14 days. Everyone is happy.
Scrum measures product value delivered in the game. This causes a different behavior than measuring estimates. Peer pressure on creating product value is far better than managerial pressure on correct estimation.
Just as with the daily status reports, my command and control side would like to see what everyone's estimation accuracy is, but I don't want to pay the price.
In antipation of this, I have been doing research and reading a few books. The best one out there is the book to be released by Mike Cohn this summer called "Agile Estimating and Planning" (the link to it is on the AGD website). I've previewed many of the chapters for this book and it seems very pragmatic.
However, there were great questions on how a project should allocate resources over the long-term. I'll comment more on these issues as we plan our next project and begin to work with our new publishers. Each publisher has their own set of expectations.
The bottom line is that we cannot trust the validity of long-term detailed scheduling. Most of us know this and do some form of granular time-boxing anyways. So I don't believe that Agile planning is that different. What is different is to assign a value to each element of your backlog to help adjust the order in which backlog elements are implemented. Also, identifying critical chains in the dev project is important.
More to come....
Saturday, March 12, 2005
It was a lot of material to present in a single hour. If I do the presentation next year I'd like to expand it to two hours. I'd expand some of the presentation areas (mostly to do with planning) and leave at least 45 minutes for Q&A.
This blog was created to answer any questions that have come up from the presentation and to develop ideas for further developing a future version.