Tuesday, December 18, 2007
This started as just another post following numerous conversations at the IGDA Leadership Forum last month, but became too large (I never like to read a 4000 word blog entry!)
Publishers and developers are really struggling with how to work together to best leverage Scrum. There seem to be some common misconceptions and patterns of problems that I wanted to address. I'll also be following up here with more detail on some specific things I cut from the article for clarity.
Thanks to Mike Cohn and Daniel Cook for their feedback on early drafts.
Wednesday, December 12, 2007
Why Extreme Programming?
Scrum is the most popular agile methodology used by game developers today. Its practices are simple and can be used with cross-functional game teams. However, for all the great practices of Scrum, it has no engineering practices to speak of, by design. While this is fine to start with, many teams using agile soon find that their original engineering practices have a hard time keeping up with the changing requirements. These teams may turn to the practices of Extreme Programming (XP) for help.
XP was a methodology developed after Scrum which adopted many of the Scrum practices. Iterations, backlogs and customer reviews, though slightly different, aren't hard to integrate into Scrum teams used to these concepts.
XP introduces new practices for programmers. Among them are the practices of Test Driven Development (TDD) and pair programming. This article focuses on TDD and the practice of pair programming since it is the most controversial and misunderstood XP practice.
First I'll start with the benefits of TDD which will help explain the reasons behind pair programming.
The Benefit of Test Driven Development
The major benefit of XP is derived from the TDD practices. There are lots of great articles about TDD out there, so I'll just give an overview. TDD practices consist of writing a number of unit tests for every function you introduce. Each unit test will exercise the function being written by passing in data and testing what the function returns or changes. An example of this would be a function that sets the health of the player in a game. If your game design defines 100 as full health and 0 as dead, your unit tests would set and test valid and min/max parameters and check to make sure those values were assigned by the function. Other unit tests would try to assign invalid numbers (above 100 or below 0) and test to make sure the function handled those bad values correctly.
If you follow the strict TDD practices, the tests are written before you write the logic of the function that will allow those tests to pass, so the tests will actually fail at first. I can't say that many of us do this all the time, but it does create better results so it's something to encourage.
Once all the tests pass, the code is checked in. Code is checked in quite frequently when the team is using TDD, sometimes every hour or two per pair. The requires a server that takes all the changes as they are checked in and runs all the unit tests to insure that no changes have broken the code. This server is called a Continuous Integration Server (CIS). By running all the unit tests, it catches over 90% of the problems that commits usually cause. When a build passes all the unit tests, then the CIS lets everyone know that it is OK to synchronize. When a submission breaks a Unit Test, the CIS lets everyone know that it is broken. It then becomes the team's job to fix that problem. Since the culprit programmer who checked in the error is easy to identify, they are usually the one who gets to fix the problem.
There are a number of very big benefits that this provides to the team and project. As a project grows, the number of unit tests grows into the thousands. These thousands of tests continue to catch a great deal of errors by automated servers that would otherwise have to be caught by people. These tests also create a safety net for refactoring and other large changes that an emerging design will require.
Catching bugs as quickly as possible allows those bugs to be fixed as quickly and inexpensively as possible. The code which created the bug is still fresh in the mind of the programmer who wrote it so they, or another programmer assigned to fix the bug, won't have to relearn the code months after it was written. Also there is the danger that in fixing the bug, the programmer may be changing the behavior of the system that may produce other bugs. By catching these problems early, we prevent the future code from becoming dependent on a flawed foundation.
One of XP's philosophical foundations is that the programmers create the absolute minimal amount of functionality to deliver what the customers request every iteration. For example, the customer wants to see one AI character walking around the environment. Most programmers will want to architect an AI management system that will handle dozens of AI characters because they "know that the game will need it sometime in the future". With XP, you don't do this. You write the code as if one AI character is the only one that will be needed. When the customer asks for more than one AI character in a future iteration, you might then introduce an AI manager and refactor the original code. Although this might represent more work up front, you will usually produce a better AI manager this way.
A major benefit of TDD is that it supports the concept of this "constant refactoring" of the code base to support this behavior. There are a number of reasons for this:
- Systems created from refactoring often match their requirements more closely.
- Refactored code has a much higher quality.
TDD is very useful and in fact is not a difficult practice for programmers to adopt. In my experience, if a programmer tries TDD for awhile, the practice of writing unit tests becomes second nature. The practice of refactoring takes longer to adjust to. Programmers resist refactoring unless it is necessary. This reinforces the mindset of writing code "the right way, the first time" which leads to a more brittle codebase that cannot support iteration as easily.
This brings us to the practice of pair programming. Pair programming can be thought of as a continual peer review. Two programmers sit at a workstation. One types in code while the other watches them and provides input on the problem they are both solving.
This practice creates a great deal of fear at first. These are some of the concerns:
- "Our programmers will get half the work done"
- "I do my best work when I am focused and not interrupted"
- "Code ownership will be destroyed, which is bad"
You can use studies and statistics to argue these fears are unfounded, but we're dealing with people's personal workspace and work habits here so things can get rather emotional and subjective on the topic of pair programming.
There are benefits from pair programming to the team:
- Spreading Knowledge.
- Assures that you'll get the best out of TDD
- Eliminates many bottlenecks caused by code ownership
- Creates good standards and practices automatically
- Focuses programmers on programming
Pair programming isn't about one person typing and the other watching. It's more of an ongoing conversation about the problem the pair are trying to solve and the best way to solve it. Given one problem and two separate programmers, you'll often produce two separate results. If you were to compare these results, you might find that each solution had strengths and weaknesses. This is because the knowledge of each programmer does not entire overlap with the other. The dialog that occurs with pair programming helps to share knowledge and experience widely and quickly.
While this is good for experienced programmers, it is an outstanding benefit for bringing new programmers up to speed and mentoring entry level programmers. Pairing will bring a new programmer up to speed in half the time and will eliminate many of the bad coding practices they may be bringing to your shop.
Assures that you'll get the best out of TDD
TDD requires that comprehensive tests are written for every function. This task is made easier by pairing. First, it's in our nature to occasionally slack off on writing the tests. From time to time, the partner will remind you to write the proper test or take over the keyboard if you are not fully motivated. Secondly, it's common to have one programmer write the tests and the other write the function that will pass the tests. Although this doesn't become a competition between the two, it almost always insures better test coverage. When the same programmer writes both the test and function, then they may not consider the entire range of tests based on their assumptions. As the saying goes "two heads are better than one". This definitely applies to TDD.
Eliminates many bottlenecks caused by code ownership
How many times have you been concerned about a key programmer leaving the company in mid project or getting hit by the proverbial bus that seems to be driving around hunting down good programmers? Pairing solves some of this by having two programmers on each problem at all times. Even if you are lucky enough not to lose key programmers, they are often busy on other tasks to instantly solve every critical problem that comes along. Pairing smooths out many such constraints.
Creates good standards and practices automatically
How many times have you discovered in Alpha that one of your programmers has written thousands of lines of poor quality code that you depend on? We generally try to solve this problem by defining "coding standards" and conducting peer reviews. Coding standards are often hard to enforce and are usually ignored over time. Peer reviews of code are a great practice, but they usually suffer from not being applied consistently and are often too late. Think of pair programming as a "continuous peer review" practice. It catches many bad coding practices very early. As the pairs mix, a company coding standard will emerge that is improved daily. It doesn't need to be written down because it is documented in the code and in the heads of every programmer.
Focuses programmers on programming
When programmers start pairing, they usually discover that the first few days are exhausting. The reason is that they do nothing but focus on the problem the entire day. Mail isn't read at the pair station. The web isn't surfed. We have shared email stations setup for when a programmer wants to take a break and catch up on mail. You never realize how much of a distraction those things were until you pair. Not everyone agrees this is good, but for those that just want to be at work for eight hours it generally seen as a positive thing.
Pairing: All or Nothing?
We don’t enforce 100% pair programming, but have done it long enough that it is pretty much second nature. If we abandoned pair programming I would want to make sure that we were still seeing the benefit above.
Pair Programming Problems
There are some problems to watch out for with pair-programming
- Pair chemistry
- Pairing very junior people with very senior people
- Hiring issues
Some programmers do not make good pairs. The chemistry does not work and you can not force it. For us the pairs are usually self selecting, so things work out. In more rare cases some programmers can't pair with anyone. Any team large enough that is switching over to a pair programming practice will have some programmers that cannot pair. It's OK to make exceptions for these people to program outside of pairs, but they will still need a peer review of their work before they commit. They usually will do some pairing as time goes by and may even switch over to do it all the time. You just need to give them time and not force it.
Pairing very junior people with very senior people
Pairing between the most and least experienced programmers is not best. You'll usually have the senior programmer doing all the work at a pace which the junior programmer can not keep up with. Matching junior level programmers with mid-level programmers is better.
You want to make sure that every programming candidate for hire knows that they are interviewing for a job that includes XP practices and what that entails. We include a one hour pair programming exercise with each candidate in the later stages of our hiring process. This does two things. It is a great tool for evaluating how well the candidate will do in a pair situation where communication is critical. It also gives the candidate an exposure to what they are in for if they accept an offer. A small percentage of candidates will admit that pairing is not for them after this and opt out of consideration for the job. This is best for them and for you.
I'm often asked about how much XP has improved productivity. We haven’t measured the changes in productivity mainly because we implemented XP during a transition to a new engine and set of consoles. It was very clear that pace of new features slowed down from this start. This corresponds to measurements outside our industry have shown that a pair of programmers using TDD are about 1.5 times as fast as one separate programmer. The additional benefits clearly put XP/TDD well above the break-even level of productivity especially for games:
90+% stability in new builds at all times. This adds to the productivity of designers and artists
Post alpha debugging needs are vastly reduced. We spent a lot more time tuning than debugging
Iterative methods like Scrum tend to cause more bugs due to the higher level of change. TDD helps addresses that.
The thing that worked for us in introducing XP was to have one team take it on to either prove the benefits and dispel the fears or to show there was no value. The team was composed of some of the more open minded programmers that were also very influential opinion holders in the company. After their initial success, everyone else wanted to try XP and prove that they could be successful with it as well. It’s good to note that our programmers will still write unit tests and refactor on their own personal programming projects.
TDD is the most important beneficial element of XP, but I’m not convinced you get the full benefits without pair programming. It continues to be a touchy subject among programmers and there is a lot of opinions about it from people that haven't tried it (or tried it for a few days). All I can say is that it does work under some circumstances, but I can't give you solid figures about how much things like stability, code quality and innovation improved since we switched. As with any practice, you need to experiment with it, modify it for your needs and keep your eye on the principles you establish for your own products and teams.
Sunday, November 11, 2007
Trent Oster's presentation Agile Implementation was very informative.
I've posted my slides from the Cross-Discipline Collaboration presentation.
Friday, November 09, 2007
Every one of the ~six talks I have been to has raised the topic of agile in a positive light. One of the most encouraging comments that I have heard was from Trent Oster from BioWare who said that the internal BioWare teams using Scrum show a 2 to 2.5 times improved productivity.
The "Leadership in an Agile Environment" roundtable was very crowded and lively. It was great to hear that their are many common solutions to the common problems that face agile adopters (mostly Scrum). Among those are publisher relationships and the confusion about long-term planning. I'll revisit these two topics in detail here soon.
One interesting thing to hear about is the topic of how much a team should change the Scrum practices up front. I hear a lot of comments like "we're doing a hybrid of Scrum and Waterfall". This seems to be a necessary phase for many company environments, but to me the jury is still out on how effective this is in the long run versus adopting full Scrum practices from the start and modifying them after they have been mastered.
- The lack of full Scrum adoption seems to cause some disgruntlement among the people who are enthusiastic about it.
- Many teams adopting Scrum don't send at least one person through certification.
- Teams that adopt full Scrum from the start have a great deal of success, but it requires someone in charge to champion the cause.
- How design and art fit into Scrum is a point of contention.
- "New Scrum fanatics" can turn people off with their fanaticism.
- Some folks on the team firmly oppose agile from the start. "It's the latest management fad" they say.
- Adopting Scrum halfway into a project creates lots of problems, but everyone who experienced these problems are excited to apply Scrum to the next new project.
Well done IGDA!
Thursday, November 08, 2007
I'm hosting a roundtable called The Role of Leadership in an Agile Environment and presenting a session called Cross Discipline Team Collaboration on Friday.
More to come...
Wednesday, October 10, 2007
- Document everything out up front so we don't have to worry so much about it later.
- Schedule out all the work that needs to be done to insure that we know what the cost, schedule and scope is.
- Implement process rules. If the rules are followed, everything will work out.
One of my favorite Ken Schwaber phrases is "Scrum is about making things visible so you can make common sense decisions". We often view processes as doing more for us, maybe solving our problems. This certainly isn't the case with agile.
Examples of the practices being 'practiced', but something is wrong:
- We might be meeting everyday, but if there are no impediments being reported and the build is broken.
- The builds are stable, the Sprints are all successful, but the game is not fun.
- We're pair programming and writing unit tests, but the continuous integration server is showing errors half the time.
One of the Agile Manifesto values is "Individuals and interactions over processes and tools". I originally took a more shallow view of this value to mean "agile practices over non-agile processes and tools". Later on I realized that the "processes" part could also include the agile label as well.
A champion is the first to be educated about agile practices. They are enthused. They see the light at the end of the tunnel of adopting the new practices. This is great, but the problem lies in how things go from there. Too many times the buy in of agile is harmed by the focus on the process over individuals. People resist change and when you tell them to change "because Scrum says to do it this way" you lose them. Agile gets labeled and resisted. The champion becomes more resisted and can be labeled as well. Their reward for trying to improve the way things are done is frustration.
A champion has to learn to be a coach. They have to communicate that the road of adopting agile is about constantly improving the product and how people work together one step at a time, and introduce some practices that can help them take steps on the road. They need to realize that there is no "light at end of the tunnel". It's all about making things brighter from day one and never stopping. The goal is about improving the product and the way we work together, not "being agile".
Wednesday, September 12, 2007
We've been applying some lean principles to game production recently during production. The results have been great and I have been returning to this book time and time again.
I've been reading about Lean for several years. It doesn't have as much of a defined set of practices like Scrum or XP. It has a very solid set of principles that can drive a wide range of practices over any type of product development and manufacturing effort. This made it particularly appealing for game development. Developing a game involves not only product development but also a production line effort to create the content for 8-20 hours of gameplay.
I’ve enjoyed reading anything that Mary and Tom Poppendieck write or say about Lean. Their writing is dense with value but easy to read. Their lean books tie in all the principles with great examples and historical perspective.
I’ll go over the main sections of the book. Although the book addresses software development, there is a lot there for people who want to work with content creation as well (producers, art and design leads, etc).
History of Lean
The book starts with the historical background of Lean in manufacturing, how the efficiencies created in the United States to ramp up war production met the philosophy of Sakichi Toyoda and his decedents as their family business evolved into Toyota Motors. The resulting Toyota Production System created a “just-in-time” workflow and “autonomation” or “automation with a human touch”. Toyota has extended this system to their product development side as well. The result is that Toyota, using their development and production systems, has become the world’s most successful car manufacturer.
Coined “Lean” in 1990, many of Toyota’s methods have been copied world-wide. There have been a number of books about the "Toyota Production System".
The chapter starts with a definition of principles: “principles are underlying truths that don’t change over time or space, while practices are the application of principles to a particular situation”.
Any book on “Implementing Lean” has to start with the principles. Mary and Tom define seven:
- Eliminate Waste - Not only work done wrong, but work that shouldn't have been done at all.
- Build Quality In - Don't fix it at the end. Build quality in as you create features.
- Create Knowledge - In your organization, about your product and your market.
- Defer Commitment - Make irreversible decisions as late as you can. Keep your options open.
- Deliver Fast - Give your customers something quickly. Continually iterate.
- Respect People - The people building the product know best how to improve the process. Create a culture where continual improvement comes from everyone.
- Optimize the Whole - Local optimization actually makes the whole less effective. Use tools such as "Value Stream Maps" to visualize how everything flows and optimize there.
How do you align your development progress and decision making closely with the value you are delivering to your customer? This chapter has a number of stories about companies and products that continually succeed at delivering value. Towards the end of the chapter it touches on leadership, team and development principles for doing this.
This chapter is one of the core chapters of the book. It covers the seven wastes as defined by the Toyota Production System (and their parallels in software development) and introduces "Value Stream Maps" which are an effective tool in mapping out your process and identifying waste.
You'd think that "waste" is pretty easy to define, but the seven wastes show that it can come in many forms. Focusing on waste and eliminating it is the core of Lean.
Speed is about delivering fast. It is defined by the absence of waste. If you practice Scrum, this chapter is an especially good read to see how the underlying theory of Scrum and Lean have the same roots.
The book Peopleware by Tom Demarco is one of the bibles that anyone working on games should read. This chapter reflects much of the wisdom and common-sense of Peopleware as applied to Lean.
One of the most valuable assets that you have is knowledge:
- Knowledge of your technology
- Knowledge of your tools and process
- Knowledge of your market
- Knowledge of what you are making
How is quality built into a lean development effort? It's built by addressing it continually using an iterative process and by being disciplined in doing the extra work required.
This chapter starts with the example of the Polaris Submarine Project. The Sputnik launch convinced the US that the Soviet submarine missile threat was imminent. As a result the US successfully moved a deadline to build the first US missile carrying submarine from 9 years to 2.5 years. How was this done? By abandoning the big plan and then using vertical slices and iteratively prioritizing features that the customer (the US) valued the most. Although the Admiral in charge of the project knew nothing about Lean, he applied many of the same principles to the project. Unfortunately these lessons were lost on most subsequent defense projects and are still not applied today.
The chapter discussed some of the Japanese practices that have been adopted for quality and how team discipline is critical to making sure that quality is part of the daily effort. Once again as you read about this you see the strong roots of practices in Scrum and XP.
This chapter addresses how the lean organization works with partner organizations (such as Boeing and its suppliers). The information here was interesting when you apply it to how the game industry is using more and more outsourcing. The best point made was that the value stream needs be simplified as much as possible before outsourcing parts of it (or all of it). It seems as though the game industry outsources a lot of work that is done the wrong way. If you address the value stream first, you may not need to outsource at all.
The final chapter summarizes the book, takes a look at some of the other initiatives that have been used (Six Sigma and the Theory of Constraints) and ends with some solid advice on how to start using Lean.
I'd highly recommend this book to anyone who has been using agile to develop a video game. The principles of Lean are directly applicable to all aspects of development and production.
As we explore fitting game development under the agile umbrella, we find that different practices have different coverage. Scrum has a great set of multidisciplinary practices that cover preproduction very well, but may not have complete coverage for production. XP practices are great for all of the development cycle, but only for the programmers on the team. Lean inspired practices for preproduction and production hold the potential of covering much of what we may lack in content production. It makes sense based on the production roots of Lean with Toyota.
Wednesday, September 05, 2007
The team can use the backlog to estimate if they are going to meet the goals based on the velocity (slope) of the Burndown and react as early as possible if there is a threat to meeting their commitment.
- The team shows consistent lower velocity in value added to the feature/game.
- The technical/design debt of the team can grow. Bugs, loose ends, etc are in full view.
- Progress of each story towards being "done".
- Dependencies and handoffs within the team.
- When we are running short of time (as early as possible) to finish everything.
- Initial concept - 1 days
- Design modular pass - 3 days
- Art pass - 3 days
- Design tuning - 2 days
- Art fine pass - 1 day
- What stage are we currently working in for a story?
- How are we tracking for total time and stage time?
Monday, September 03, 2007
“Principles are underlying truths that don’t change over time or space, while practices are the application of principles to a particular situation”.
Agile Game Development preserves the principles of agile while seeking better practices to apply them.
Success or failure with agile seems to be based on how the principles are maintained.
Saturday, September 01, 2007
We were having a discussion the other day about how it seems that most great games which come out go through some form of crisis or independence from deadlines. One development story was about MechWarrior 2. The team had struggled to find the gameplay to allow funding to continue, but at one point the game was canceled by an executive on a Friday. The team struggled to finish a pass at gameplay over the weekend and fortunately succeeded in overturning the decision Monday morning.
It seems that every breakthrough game had a similar chaotic history or had a commitment by the developers to “release it when it’s ‘done’” (which was longer than the publisher or public liked).
No publisher in the world is going to sign up for a “release it when it’s ‘done’” development deal with anyone besides an Id Software or some other similar developer who can share the risk and who has a proven track record. It’s reasonable considering the risk, but what potential great titles like MW2 didn’t make it?
The Publisher Business Model Rules All
Publishers are like any other large business. They have shareholders that expect returns on their investment and a board of directors that directly answer to those shareholders. The board demands a business plan from their executives that show perhaps a five year plan with lots of promise of profit. To create this plan, the executives develop a portfolio of game products with an estimated profit/loss (P&L) statement for each. Obviously each future game in the portfolio has a positive P&L or it wouldn’t in the portfolio. Unfortunately a large portion of games released do not make a profit. We are still a hit driven industry, but hits can be a bit difficult to predict. Hit franchises give you the best chance (Madden, WoW, etc), but many franchises fade over time.
The portfolio should show a good mix of proven franchise titles and riskier titles that are potential new franchises (new licenses or new intellectual properties (IP)). The problem is that riskier titles won’t get a big financial bet. In order to get a positive P&L you have to compare the predicted sales with the cost of development (and marketing, etc). No one in their right mind would predict huge sales for a brand new license or IP. So to keep the P&L positive, the dev cost has to be low.
Developing the Riskier New Franchises
So there are a number of new riskier titles in every publisher’s portfolio. Unfortunately they can often be cast in the same date driven mold as the proven franchise development projects. They needed to be treated differently.
Risky new titles should be developed with the P&L in mind, not the portfolio. The portfolio is a wish list used to predict the market and the marketing flow. It’s a poor tool to apply to new titles. Like Design Documents, if anyone takes the time to look back a few years at original portfolio plans; they bears little resemblance to what really happened.
To maximize the units sold and minimize the cost of development, development of these titles needs to pursue an effort that maximizes efficiency, is quality focused and minimize risk to the publisher.
Research has shown that there is a “sweet spot” in the number of people working on a task; fewer or more people than this sweet spot and the cost of development rises. This is the basis of the Mythical Man Month and Scrum practices of what team sizes should be. Brooke’s Law is especially true of the game development industry where adding more people to a game development project in pre-production not only has little effect on the ship date but can actually slow down the team with communication overhead and loss of focus.There are many ways to minimize production time (outsourcing, reuse, etc) by improving efficiency and partitioning, but pre-production is a highly creative and iterative process that can slowed down by applying the same “fixes”.
Better games can be made by having small creative (inexpensive) teams iterating over the possible gameplay mechanics.
The first factor in quality is the team. Talent and teamwork is the key to success. How do you know whether the team can deliver? Iterate. Have the team demonstrate their progress every few weeks. Participate in the direction that the game is taking. Marketing should spend time in reviews and planning as well.
This approach fits well with the kill gates approach to seeding many ideas and allowing only the best to fully grow. This sounds harsh to some, but it’s better to kill a bad game early than to spend years working on it to see it fail in the market. Also, impeding cancellation of your project can have a great motivation for the team to prove the games value, as with MechWarrior 2.Incentive to Take Risks
Based on the current business model, it's difficult for the developer and the publisher to adopt a pre-production model that allows for quick cancellation if the game is going nowhere. Developers want predictable, long-term cash flow. Publishers want to keep the portfolio flow going and want a sense of security from a big plan.
However examples for this models exist (see below). The important thing is to preserve the relationship and pursue ideas with a talented team.
Where does Schedule fit in?
At some point in time, it should be apparent what the core game is and what it is going to take to finish it. At this point the production side of the project can ramp up and a low-risk gold date can be predicted.
Production can’t be ramped up overnight. So every release should have an evaluation of the project in terms of the conditions for production that remain to be met. The team should be able to give a release or two advance notice (~ 3-6 months) of when production can begin. Longer notice can be given with less certainty.
Where games took 2 years to develop, they now might take longer. Teams may save some of the time by entering production with truly shippable mechanics rather than wasting time reworking assets in light of changing mechanics. It should be a lot less expensive and produce better results.
The main risk of this approach is rushing the production trigger. It’s hard enough not to fix a launch date two years in advance. Asking for a year or less commitment to the launch date may be impossible for marketing up front, but it’s what is often done in light of project delays.
Not a New Idea
None of this is new. This approach dovetails with agile very well though. As more developers adopt agile practices it will make more sense to develop games with this approach.As a member of the Nintendo Ultra-64 Dream Team, Angel Studios was exposed to this model in the mid-90s. Nintendo would discuss a game idea with us and ask us to "find the fun". They'd give us enough money to operate for three months and then come back at the end of that time to see what we found (occasionally Miyamoto would visit!). Usually we discussed the results and the high level direction for the next three months. Occasionally, if we couldn't "find the fun", the game would be canceled and an entirely new game would be started. If the game lived long enough Nintendo would tell us to finish it in six months! Unfortunately we weren't very iterative in our development approach at Angel, so it wasn't an ideal fit.
Wednesday, August 29, 2007
I agree that contract solutions for iterative development are different, but there is no reason that existing models can't be applied. In fact, Tom wrote an article about flexible milestone deliverables. He does point this out, but says that Scrum is incompatible with "hard-set objective-predetermined milestone deliverables", but this is not what the flexible milestone deliverables article is saying. In that article he says that the publisher and the developer define each milestone (past the first two) based on the the last milestone. This is an iterative contract approach! This is not incompatible with Scrum. In fact flexible milestones, used by many publishers, are a acknowledgment of the iterative nature of game development.
Scrum is a framework. For example, it doesn't define technical practices or how to handle "milestones". Practices can be added to support both. For example, many teams add XP practices like Test Driven Development to create code that can better support iterative development. Another set of practices are those defined by Mike Cohn in his book Agile Estimating and Planning. There is nothing against with defining milestones the same way you would define releases. Your product owner should pay attention to the amount of change that is introduced during the release so that the release backlog matches the milestone definition agreed to. If this is too hard to do over three months, work with the publisher to have shorter milestone intervals. I don't think many publishers would argue with receiving more regular builds.
If anything this iterative milestone approach encourages communication about the product between publisher and developer. Contracts with fixed milestone deliverables/payments discourage change and therefore communication. Both parties may agree that the game is not as fun as the original design document and contract anticipated halfway through development, but for the sake of those payments, no change is allowed (and I have never seen a contract rewritten during development).
At Agile 2007, I went to a session called Product Ownership. I was interested in any session of product owners because we have hit issues with having one product owner for a large team and have experimented with having embedded product owners in multiple teams. I was hoping to hear stories from other teams who have addressed the same problem.
The session consisted of three good presentations, but one in particular hit the mark in addressing the multiple product owner solution. The presentation was called "Effective product ownership within a multi-component project" by Mike Lowery and Marcus Evans.
The presentation covered a project being developed at the
The team immediately ran into problems having one product owner (quotes from the proceedings in italics):
- The Product Owner was so busy that he was not available to make timely decisions.
- Even though the Product Owner was manically busy, he never actively empowered the teams to make their own decisions. This lead to an interesting shift in development. Some teams were frozen by this and development slowed to a crawl. Other teams did what they felt like and moved the development forward only to have work scrapped when the rumors stopped and the Product Owner finally gave direction.
- Product backlogs were not prioritized so teams started to build items in the order that they preferred and moved unattractive items to the bottom of their product backlog...
- Whenever any attempts were made to prioritize the product backlogs, the Product Owner struggled because the items on the list were only understandable by the teams who wrote them.
- The number of scope meetings rose to fever pitch, as what was in and out of scope was never clear and the Product Owner was never available.
The team went through a few changes to the product owner role, creating two product owners and eventually moving to having each team appoint an embedded product owner (
- Clear product ownership.
- Definite project direction.
- Questions were answered quickly.
- Delivery teams had planned thought out product backlogs.
- Work picked up pace and we were back to delivering new functionality each iteration.
- The in-team Product Owners became subject matter experts.
- Each team put itself into a specialist silo.
- Team direction was not always business direction.
- Product backlog items fell through the gaps between teams.
The first concern raised about this (and we raised it ourselves) was how to avoid silos for each team and a lack of vision for the product as a whole. This is addressed by maintaining the central, or core, team with a project product owner.
The Product Owner from the central team does not give direction on individual features and rather is there to ensure that the overall business direction of the project is maintained. They also ensure that things do not fall through the perceived cracks between the teams, and that dependencies and impacts are captured mapped and coordinated. The in-team Product Owners were retained to act as product specialists more than pure Product Owners who set priorities.
The role of the core product owner is to identify the epic (large) stories and their priorities for the project.
This has worked well for the
Sunday, August 05, 2007
Sunday, June 17, 2007
A Disturbing Assumption
I occasionally come across comments like this one:
I don’t agree with this prediction either way and we have tried to emphasize this in the presentations:
Agile is not a silver bullet
Some of the key points we stated were:
- There is no “right way” to make a game.
- The talent of your team is the main factor in the quality of the game.
- Agile has benefits:
- Reducing wasted effort
- Finding the fun first
I’m not saying:
- Using agile makes you a hit factory
- That High Moon has any greater insight than anyone else using agile
- We’ve just been talking about it for a few years.
- Waterfall will always fail.
As for High Moon, I’m enjoying myself. Its fun, challenging and we are improving how we make games every day. Will the next game be a hit? I don’t know. It has the strong potential but I’ve given up predicting success solely based on the development side alone. I’ve never predicted it based on the process used to create it!
Let me back that up with a look at a past project I worked on.
One game I worked on was Midnight Club at Angel Studios. It was not a game developed using agile practices which makes for good comparison.
- Fixed price, fixed schedule development (it was a PS2 launch title).
- We shipped on time and budget
- The original scope included six cities
- We shipped with two
- The last four months of development was tough.
- 72 hour weeks for most.
- The game sold 2.5 million units
- It was a success for the team and the customer
How did it succeed?
The keys to success were:
- It was a great team, especially with the technical leadership of Dave Etherton and design leadership of Fred Marcus.
- The team largely consisted of the Midtown Madness team. Vision wasn’t a problem.
- Marketing of Rockstar
- These guys were amazing. Who else would pay to close down
Times Squarefor a night to get box cover shots?
- Closely tied with talent and experience, but there is benefit to knowing your mature and adoptable technology base from the start. Adopting the PS2 was not easy though.
How would agile have helped Midnight Club?
Agile could have helped mitigate the crunch at the end. Many of us were truly burned out after MC1 and a few good people left. I don’t know if we could have turned out a third city, but dropping that city late in development had a major impact on the flow of the game. Knowing we were going to ship with two cities earlier would have reduced the effort wasted on the third and allowed the designers to adapt the gameplay for two cities.
There are two key points to summarize
- We succeeded with waterfall.
- Agile could have helped significantly with crunch and somewhat with gameplay.
Waterfall is not a blank bullet
Waterfall has been rejected largely due to the higher percentage of projects that using it fail (miss budget, quality or schedule goals or are canceled outright). That percentage is not 100% though. Process contributes to but does not determine success.
The point of agile means you shouldn’t accept or reject it blindly
Agile methods use an iterative approach. You are constantly in an “inspect and adapt” cycle. This applies to products, teams, whole companies and even the process itself!
Frameworks such as Scrum or XP suggest practices that support the “inspect and adapt” cycle. What you do with the results of your inspection is up to you. If you find yourself inspecting how teams work together, including everyone in how the operations of the company can be constantly improved and trying to honestly ask yourself if the projects are headed in the right direction on a regular basis, then your shouldn’t care about High Moon’s games!
Scrum is self modifying in nature. If you aren’t achieving the results, then change the process! Just try to preserve the principles though. Too many scrum adopters remove or ignore the principle of change to their detriment.
High Moon’s Next Game
Regardless of the success of failure of our next game, I believe that we are better off using agile to develop it.
If our next game is a hit, does that mean you can do the same by just adopting our practices? No! It means that we had the talent, technology and marketing to make a successful game. If the game fails, we will not blame our process, but ourselves.
The goal is to compete with each other on the basis of talent and not the illusion that we are competing on the basis of who has management that can make their people work longer hours.
Building your studio culture to never be satisfied and to improve your games and how you make games is the ultimate goal. When I read articles like this, I get anxious to help build a culture like that. The irony is that you can't impose this kind of culture. It has to be grown though trust and ownership at every level. These are principles which were used to create agile. Agile can contribute to culture, but it's just a part.
Thursday, June 14, 2007
- Iterations that produce potentially shippable versions of the game.
- Test Driven Development which surrounds every function of code with unit tests that execute many times during the day.
- Dedicated build servers for continuous integration and automated testing (unit and functional tests).
- Stop-the-line culture and tools. When a check-in (code or assets) break the build, every developer knows it because a desktop icon turns red and a jingle plays (such as the Swedish Chef singing his theme song). If your commit was among the last ones, you are pestered about fixing it. We also dedicate programmers, on a revolving basis, to tracking down and fixing crashes. When the game crashes, regardless of the platform it is running on, an email with crash details is sent off and a core dump saved to a network drive. The programmers assigned that day to monitor such crashes jump on the data and try to find the cause as quickly as possible.
- Iteration - The teams adjust to delivering value every iteration.
- Engineering - Your engineering practices adjust to support iteration.
- Ownership - Teams act like they own their day-to-day work.
Sunday, June 10, 2007
Sprints are valuable to show a potentially releasable feature or version of the game to customers. Often who those customers are makes a big difference in the focus of the sprint.
There are three main types of customers who can review a sprint. I'll list them in order of increasing importance:
These are people around the studio, including studio leadership. Often the teams that get the most benefit from studio customers are the functional teams, those teams that are creating tools and underlying technology for the feature or production teams to use.
These are folks from the publisher. Getting them to participate in the review every sprint can be impossible, but it’s valuable. They haven’t seen the game and so they are less susceptible to the Ugly Child Syndrome. Watch out for the
It’s great to focus test a sprint build. It’s even better to focus the sprint on the focus test. In other words do the following:
- Let the team know that the sprint build will be focus tested from the start.
- Build the stories around focus testing
- Have them participate in the focus testing
Focus testers make the best customers in my opinion.
- They evaluate just the game
- The information you gather from them is more honest. They don’t have to worry about hurting the feelings of the dev team especially if you conduct the tests correctly.
- They are your ultimate customers!
- Dev teams don’t fear them.
The last point is key. As a “Chief Technology Officer”, I make a lousy customer. I influence salaries, bonuses, promotions, etc in my role. I would love to take that hat off and put the “customer trying to represent the consumer” hat on and discuss the game or feature objectively. However the two roles blur and communication can’t be that fearless.
Note: before the comments pour in on the comment: “I influence salaries, bonuses, promotions, etc in my role” I would like to point out that creating a truly agile incentive and reward system is one of the hardest challenges. It’s not only the practices that need to be changed but some of the deep cultural biases.
When Microsoft was focus testing Midtown Madness in 1999, they were just starting to develop this capability for games. They’d stuff 20 people at a time into a small room with PCs and have them fill out questionnaires. It was a far cry from what they do now, but the value was huge. Conduct some form of focus testing early. It could be as simple as recruiting four gamers at your local GameStop to come over one evening, eat pizza and play the game. You can build up the process and equipment for focus testing over time (neutral questioning, two sided mirror rooms, etc) later (with all the royalties).
Definitions that I use for defining the main types of Scrum teams that contribute to a project.
The team which owns the game and insures that integration of features, functions and production meet the needs and vision for the game.
Teams that focus on vertical slices of core gameplay features for a game.
Teams that focus on horizontal slices of the game (graphics, animation, audio, pipeline) in support of the feature or product teams.
Teams that fill production requirements of the product teams. Very similar in spirit to outsourcing in that iteration is a minimum. Assets are created for known features.
Our child was something only we could love.
It’s hard to tell a feature team that their child is ugly, but it is exactly what is needed. The sooner the team knows it, the easier it is to turn it around.
This is the value of having external (publisher or focus test) people who speak their minds play the sprint review build. Customers that see progress on a daily basis start to see beauty where the buyer wont.
Sunday, April 22, 2007
I'm very interested in what will grow out of this venture. They've started a blog to document their experience. Let's watch!
Sunday, April 08, 2007
Pre-Production & Production
One of the major differences with using agile for game development is the separation of activities during pre-production and production. In the past I have shown this using a version of Ralph Stacey's Agreement and Certainty Matrix:
This argued for more iterative and exploratory development during preproduction. One of the goals of preproduction would be to gain certainty of the game's technology and come to agreement on the requirements for all production assets. This would likely be done by creating a number of final quality levels and characters, for example, that would demonstrate the scope of the full game.
Once in the "simple" area of certainty and agreement, you can apply production methodologies (some prefer "lean production", others more traditional waterfall like methods) to complete the set of assets to ship the game. If you are using Scrum, then you might change some of the practices in production in the following way:
- More is scheduled and preplanned. Less planning occurs in Sprint planning.
- Smaller backlog, bigger Gantt charts
- Sprints and Daily Scrums address support for the production crew.
Where do Prototypes Fit In?
When you are working on new genre, engine or platform, you're actually in the "anarchy" area. This area is so fluid and uncertain in nature that the standard Scrum practices need tweaking here as they do in the "simple" area.
Prototypes of what you want to do in a new genre, engine or platform environment would live in the state of "anarchy". The main value of prototype iterations is knowledge, so prototype efforts differ from pre-production because you're more likely to throw away what you've tried during the last iteration and try something else.
Scrumming for Prototypes
So how should the Scrum practices change for prototypes? There are two issues that need to be addressed.
The first issue is that even trying to plan for a two week Sprint cannot be carried out with a great deal of certainty. When you can't do that, you can't make a commitment to a well defined Sprint backlog. The team needs to be able to commit to take ownership.
The other issue is focusing on building knowledge through focused discovery. We don't want the team going off hellbent on some open-ended, ill defined goal that could take up too much time.
The answer to this is use more spikes. A spike is a time boxed experiment which is meant to produce knowledge. This knowledge is used to improve technical certainty or requirements agreement.
Spikes usually refer to tasks which produce knowledge for user stories, but in prototypes we use them to refer more to stories themselves or even the entire Sprint.
So getting back the Scrum practices our team has tried a few things that work during prototypes:
- More fuzzy stories for the Sprint
- More exploratory
- Conditions of Satisfaction that are more subjective (e.g. is it fun?)
- Can be met early in the Sprint, with room left for continual improvement at the back end of the Sprint (iteration within the iteration).
- Full team ownership
- The product owner for the feature is on the team doing work and the entire team is creating new ideas (the fate of the PO and team are shared).
- Larger time-boxed tasks
- e.g. "iterate with game designers for four days" (you have four days to make it fun).
- Daily Scrum is different
- We start by playing the prototype (let's see where we are at).
- New tasks introduced (inspect and adapt)
- Time-boxed work is broken down (we get specific when we have the knowledge).
- Eliminating the burn-down chart
- When almost everything is time-boxed, what's the point?
Hey! That's not Scrum Anymore!
Yeah it is. See my last entry.
Saturday, April 07, 2007
I often read DanC's "Lost Garden" articles and have been thinking about this one written a month ago that struck a chord. "Kill Gates" are a brilliant way of approaching a portfolio of games for a publisher and a set of key features for a single game.
The idea is that a team can use an iteration-feedback cycle to rapidly converge on the best game possible. Conceptually this is very straightforward and makes sense. Teams adopting iterative practices, such as those with Scrum, have typically seen immediate improvements. However there is a typical stumbling block that occurs with many teams that prevent the full benefit of an iterative approach that DanC describes. This stumbling block has to do with the role of "Product Owner".
In Scrum, the Product Owner (PO) is a customer who prioritizes the set of features desired for the game into the "Product Backlog". Each iteration the team looks at this backlog and selects a number of higher priority stories that they commit to completing. They come back two to four weeks later and show the game to the PO.
The stumbling block is that many teams often don't have the best possible product ownership in place. There are a number of reasons for this:
1. PO experience. Mike Cohn and Ken Schwaber have been conducting Product Owner Certification Training classes to address this issue.
2. No PO in place. Teams can be their own worst PO. Pet features and individual interests can prevent the objectivity that a good PO can provide.
3. A PO who is too distant from the iterations and teams.
This third condition is the one that can be the most difficult to address. One of the key Agile Values is "Responding to Change Over Following a Plan". If a PO doesn't see the game often enough or prioritizes based on the plan on their head, then you could be iterating towards a bad product.
PO with a Plan
The "document in the PO's head" sets very strict limits that prevents the team from responding to what is working or not. As with any other "upfront plan" based approach, it postpones the reality of what is being created and results in wasted effort (cost overrun, schedule slip and crunch).
Aligning the PO role for success
One of the major emerging differences with our implementation of Scrum (as opposed to the standard practices you start with) has to do with the role of the PO. We no longer believe in having a remote PO. The reason is that it's almost impossible for a game Scrum team to work for two weeks without input from the customer. Too much of game development involves intangible elements that defy enough definition for a team to predict their work for even a Sprint. In fact, when prototyping a new mechanic, the team will often want the PO available all the time. The teams have changed a few things to make this work:
- More Spikes where necessary
- Product Owner as a Pig
- A Hierarchy of Product Owners
How many hours will it take to make Mechanic X fun? How many tasks are required? You can probably predict a lot of the work that is required to get the mechanic working with some test assets, but you need daily, hourly and even instantaneous iteration to explore and discover the magic recipe that will entertain the player. That part cannot be easily broken down into tasks, but the team can estimate a certain portion of time that it will explore. This is called a Spike. A Spike is a time-boxed task that allows flexibility with some predictability. By trying to predict too much with more defined tasks you can actually limit yourself and go down a wrong path (it's a form of a plan with not enough knowledge). Spikes create knowledge. In fact while creating a initial prototype, half the team's time taken up with Spikes.
Product Owner as a Pig
If the team wants to explore with Spikes, they will need instant feedback. This feedback is not only cross-discipline (e.g. designer-programmer), but with a customer. The customer may own the vision of a key mechanic.but the customer is supposed to wait for two or four weeks before they can give feedback. This may not be best for the progress of the mechanic. So does the team drop the practice that the customer is supposed to let the team take ownership and commitment or make them wait? Neither choice is best under these circumstances.
Our answer was to make the customer part of the team. Typically the on site customer is a programmer, artist or designer anyways, so they can add value to a team by taking on tasks. By giving this customer the ability to prioritize the work for that team, you are also making them the PO. By making the PO a part of the team, you are creating accountability in that PO for the work being done. Accountability works wonders on how well that PO will prioritize and pay attention to cost. It works.
A Hierarchy of Product Owners
So now we've created a PO for every team creating a mechanic. This can create problems. This can cause a bunch of mechanics that aren't consistent to be created. These PO's need to work together and there needs to be one PO who is guiding the entire project's backlog. Call this the "Lead Product Owner" if you'd like.
Hey! That's not Scrum Anymore!
There's been a great deal of debate (again) about what Scrum is at the Yahoo Scrum Development mailing list recently. This time around Ken Schwaber created a separate list for those who want to carry on with the debate. The message from Ken (that I agree with) has always been that the Scrum practices are meant to be changed for each application. The agile/Scrum values and principles behind them are supposed to remain constant.
The main debate that we have had is whether the changes above take ownership and commitment away from the team. For large teams, we have found these changes to be necessary. Small teams self-organize fine, but large teams have additional issues that prevent this from happening. Mary Poppendieck uses an analogy for bands when describing the level of leadership required for different sized teams:
- Level of Leadership
- Jazz Ensemble – Set the tempo and start the piece. Rotates.
- Jazz Band – Direct the band, but not featured instruments.
- Orchestra – Direct the orchestra and interpret the piece.
Remember, this is what we came up with and may not be best for you. Don't be afraid to experiment. At worst your velocity will suffer for a Sprint and you'll change things again. Scrum is about reacting to change.