Search

Sunday, December 21, 2008

Learning about production in pre-production

Have you every been on a project with a date scheduled that says "Start production"? This is the date when the team transitions from pre-production to production and a ton of people (internal or outsourced) join the team to create the 10+ hours of gameplay to ship? The idea is that the team knows what's fun about the game and can build a ton of assets in parallel.

Did that deadline really work out? Was the team really ready to "go into production"? Most aren't!

The main goal of pre-production is to build knowledge, or learn about the game. We want to know how fun our game is, how we are going to make it and how much it is going to cost. Building this knowledge requires iterating on the areas where we lack knowledge. We choose what to work on based on a priority of the knowledge we want to build. For example, we’ll iterate on the core mechanics first because the provide the greatest knowledge of the value they provide to the game. We’ll also iterate on building characters and levels to learn how much effort and cost is required to make many more of them in production.

We often focus more on learning about the core mechanics, or the fun of the game, and not enough on how we are going to make it. This results in many projects exceeding their production budgets or schedules. What we need to do is to learn the cost of building assets during pre-production as well as learning about how fun the game is going to be. Both of these go hand-in-hand since we can’t determine the cost of production assets until we know what makes levels fun.

Let’s look at level production. Level production can easily require 50%-75% of the production budget. We need to refine our understanding of the effort to build levels during pre-production so that we do not build mechanics which inflate production costs beyond our budget. For example, a fantastic feature for the consumer would be a mechanic that allowed every object in the environment to be destructible. However implementing this may also double the effort to create levels since building destructible geometry everywhere requires far more effort. By knowing these cost implications, the Product Owner (PO) can better judge the value of adding this feature or not.

Learning about production cost is an iterative process. We begin with a range of estimates based on existing knowledge (perhaps from a previous title) and iteratively refine these estimates during pre-production. As we move forward in pre-production, we are building a vocabulary. At first we are establishing the alphabet. This may be the second-to-second game-play experiences such as battling one enemy character. We then start building words from our alphabet. These are the 10-60 second loops of gameplay, perhaps dispatching a wave of AI characters in a room. From here we build sentences, then paragraphs and chapters of larger loops of game-play.

We follow this same pattern in building levels. We don’t start by building an entire level until we build a vocabulary of rooms. We may build several different types of rooms with different experiences (e.g. lots of AI in a plain room followed by a boss in an ornate room). We don’t wait until our room vocabulary is final before we start building our level vocabulary. Building a shippable level before we have a minimum room vocabulary will waste a lot of time. This waste is seen on many milestone driven projects. Teams feel compelled to show a level of polish to their publisher when the gameplay is still undetermined. These polished milestone levels end up as either waste or debt; They are eventually thrown out or require a great deal of rework when the team learns more about the game-play.

Firing tracer rounds

Sometimes it’s valuable to demonstrate a 20 minute experience with a quick and dirty level filled with stand-in assets to build knowledge of our level goals. This in turn may influence the lower levels of our vocabulary. Another valuable test would be to create a single room with as much graphics polish as your engine can muster to help refine our visual DNA and target. Although these assets may be thrown away, their cost is easily worth the knowledge they create for your project and customer. The key is to not waste more effort than the knowledge is worth.

The PO is responsible for ROI

Since the PO owns the return-on-investment for the project, they need to demand that the team demonstrate improved knowledge of production costs refined through pre-production iterations. They do this through separate stories or conditions of satisfaction on existing stories. An example is “demonstrate the cost (in actual hours) to create the X level”.

The last pre-production release should have a level or two which demonstrates the major range of vocabulary at potentially shippable quality. This is the only way the team can demonstrate that the knowledge of production costs have been learned.

Wednesday, December 10, 2008

Scrum for artists

“Why use agile for art? When Michelangelo painted the ceiling of the Sistine Chapel he wasn’t using agile. He had a plan to paint the entire ceiling.”

These words were spoken by an artist I worked with when we first started using Scrum. Little did I realize at the time that Michelangelo might have been better off using a more agile approach to painting the ceiling. He had no idea about how to painted images on a curved and segmented ceiling. He signed a fixed price contract calling for 12 figures painted. Four years later he had painted 300 figures. There was a great deal of trial and error and false starts. It’s no wonder he referred to it as one of the worst experiences of his life.

Historical accuracy aside, the questions and impressions about how relevant agile or Scrum is for art creation remains common.

The following are common perceptions about Scrum from artists:

Scrum was created for programmers.

Scrum is largely used in software development projects, but it wasn’t created for programmers.

In fact it has no practices specifically for any discipline. Scrum was defined by studying how many different products (such as cameras and cars) were developed. It’s just as applicable for artists as it is for any other discipline.

Art production runs on a schedule. We can’t be iterative.

With video games, art and technology have to serve the gameplay mechanics together to create a great experience. We have to explore how all these components work together to create the best possible experience. Once we discover this, we can schedule making 10 to 12 hours of this experience. Exploring what we need to create before we mass produce assets is beneficial. Cross-discipline teams support this.

Artists often want to work with other artists.

This is the same with other disciplines. They speak the same language. The problem again is that our product requires an integration of all disciplines. This forces some form of collaboration eventually. Unfortunately a studio structured to separate the disciplines (though a heavily matrix management structure for example) will discourage collaboration or at least make it more difficult. If an artist is evaluated by the number of assets they can create (through promotions or salary reviews), then they will optimize their work along this line. Interrupting an artist to help with a game problem will impact the number of assets they can create, so they discourage those interruptions.

Scrum might be good for preproduction but not for production

It’s true that Scrum is better for preproduction than production. Production requires a more defined work flow that must create a fixed amount of gameplay to ship on a specific date. The Scrum practices should be modified by the team in production. However this doesn’t mean we won’t have challenges in production. The game will still continue to throw challenges at us that we don’t expect. These play havoc with the best planned schedules. Artists creating assets will still need rapid response to problems emerging that they can’t solve. They will need to continually find ways of working more effectively through improved tools and pipeline processed.

While the iterative nature of development will slow during production, the need to incrementally improve what we are making and how we are making it will never slow down.

What specific problems are we trying to solve on the art side with agile? How is this going to make better for the artist and team in general?

We need to know if we are creating the right thing and not wasting time.

Parallel development of assets and technology that the assets depend on is a traditional source of wasted effort. Engine development is often started with optimistic feature sets, performance goals and schedules. We can’t simply shift engine development to iterative development without an iterative asset creation effort. Iteration on technology requires an ongoing conversation and experimentation with what looks and works best. Every artist knows that the quality of a complex asset such as a level depends on a tradeoff between polygon count, texture quality, lighting complexity and the palette of effects available. None of these qualities is independent of the others. Some levels will require more effects than others and there will be a envelope of tradeoffs that occur to balance the overall aesthetic.
We want to build the knowledge of these tradeoffs during engine development before we commit to production. This requires daily collaboration between the engine creators and the first order customers: the artists that use the engine.

We need to have a working build.

Nothing will impact progress more than not having a build that fully works. Graphics bugs that impact the visual quality can prevent an artist from fully evaluating their work. What’s worse, addressing these problems can be put on the back-burner by a separate technology group that is solving problems more relevant to their own local needs. Cross-discipline teams are more likely to have someone on the team who can solve the problem or know someone who can.

We need faster tools and pipelines

Artists are often constrained by the limited or slow control they have over the game. There are often barriers to iteration that slow their progress down. Less iteration means lower quality. The fundamental problem is often that the programmers who have control over improving tools and pipelines are not impacted by their limitations. They don’t see how slow it is to change a texture on a game and see how it looks in the game. They are focused on the tasks that are important to their local team.

As with the build issues, having a programmer on a cross-discipline team experience the drag on the team’s velocity that poor tools and pipelines will help solve the problem. Smoothing the entire pipeline from concept to user experience will allow a better game to be made. Cross discipline teams are the best way for this to happen.

Tuesday, December 09, 2008

The role of QA on an agile team

"When I was a child we used to suck on pennies...and it was a delight!"

Every year or so I’m invited to speak at one of my son’s schools to speak at one of those parent career day events. It’s always been fun. “Video game developer” is the coolest career imaginable to these kids. I could be standing next to the astronaut-secret agent dad wearing his space suit and they would climb over him to ask me questions. It definitely scores some “dad points” with my boys.

One of the inevitable questions is “how can I be a game tester?”. They believe that the role of tester is eight hours a day of the same experience they have playing games. What could be better than that? Unfortunately I am forced to shoot down this notion. I explain to them that most jobs in the game industry are in very high demand and we get to choose the people with the best grades. Making games requires excellence in everything they are already studying. The math they may not like is what we use to calculate where Mario jumps. The literature class teaches them how to create stories which are essential to games as well. The list goes on. I merely have to point at their work on the walls of their classroom and tie this or that into work that needs to be done making games.

Getting back to the job testing…I describe the long hours with games that crash . I describe how testers often are very frustrated because they have an understanding of what the game is and have very little influence over over how the game is developed. I don’t want to discourage the future generation of testers. I just don’t want to give the impression that there is this wonderful career with a low bar of entry just waiting for them because they are “good at playing games”.

The traditional role of QA

The traditional role of QA on a game project is to pile on a project in its end stage. At this stage, the important decisions about the game have been made and the job of QA is to find and report the defects. Many of the defects are too deep in the design or architecture of the game to be addressed so late. For example, if the “shooting mechanic” just isn’t fun, it may be too late to correct it. Let’s just slap a band-aid on it (e.g. hide the poor aiming behavior with an effect).

As you may guess, the QA role attracts people that are passionate about games. The sheer number of people applying for these role gives the industry the ability to choose among the best. We should be leveraging the value these people bring far more.

The new role

Agile teams can extract more value from testing. Much of this value comes from the team’s goal of delivering a playable game every iteration and the principle that results of the iteration can influence the plan going forward. The QA role shifts to take advantage of this:
  • Rather than sitting in a pool of testers, testers are spread through teams.
  • Stories require QA action for approval before they are considered done.
  • QA’s voice is heard throughout development, not just the end.
  • QA represents the consumer.
The role of QA on an agile team has to change. The traditional approach of “give me the alpha game and I’ll give you bug reports” are gone. The tester’s role on an agile team has to grow. This starts with the tester’s participation in the Sprint planning meeting. When each story is discussed before the tasks are estimated, the tester needs to understand how that story will be approved. This usually takes the form of how the story and the conditions of satisfaction (CoS) are written.

For example, take the following story:
“As a player, when hit the jump button, my character will jump”.
The team would have a discussion about this story and may ask the following questions:
  • Will the character have jump animation? Will it need to be smoothly transition?
  • Will the character be able to jump to a higher level?
  • Can the character jump from a moving platform?
  • Can the character jump while walking or running?
These questions will help the team understand the tasks that will be necessary to implement the story and how to test that the story is actually “done”. For example, the jump story may define the CoS that result from the answer to the questions above:
  • The character will have a simple stand-in animation that will not be blended.
  • The character will be able to jump up to a ledge, or down to a lower level.
  • The character will not be able to realistically jump from a moving platform.
  • The character can jump from any starting motion, but momentum of the player will be ignored. It can only jump straight up.
With these CoS, the tester will be able to verify that the story is complete before the end of the Sprint.

The tester should also catch problems that are not part of specific CoS, but appear as part of development. For example if the test level is missing some physical geometry that allows the player to “fall through the world” the tester should raise the issue as an impediment to testing the game at the daily Scrum.

In addition to testing the game, the tester should help to insure that the team is not impacting other teams through their work. This would include:
  • Performing regression tests on team builds if major changes are about to be committed that might break some obscure part of the game.
  • Testing tools and pipeline changes that a support team might want to release to the content creators.
  • Finding ways to improve the testing process. For example, if tests are frequently encountering a bug associated with bad texture imports, the tester might raise a request to have the automated tester or even the exporters improved to catch the problem earlier.
Much of the tester’s time will be spent verifying the gameplay and offering advice on areas of improvement. This requires an ongoing dialog with members of the team.

Since the tester represents the consumer, they need to keep a “consumer eye” on the game. They should point out any issue that the consumer will notice (such as a bad texture or gameplay progress stopper).

The benefit to the value of what the team adds every Sprint will greatly benefit from having a tester on the team. Customers will get what they ask for and stories will be truly “done”.

The challenge

The tester is challenged in this new role. As mentioned above, the traditional approach doesn’t work on agile teams. The tester is required to communicate with every discipline, understand the tools and stand up and be heard. Not every tester I’ve met can do this. Some I've met are more than ready. QA has traditionally been a “gateway role” for other roles in game development, so it’s an appropriate “test of fire” for people with those skills to grow their career in game development.

Sunday, December 07, 2008

Scrum as a tool for process and cultural change

Any process can be used to make games, but no process is perfect. Even if we were to find a perfect process, the constant change in our industry would make it quickly obsolete. Scrum is not a process, it’s a framework for creating your own process. It can help any organization create transparency which enables competent leadership to react to change with common sense.

When we first adopted Scrum, every few months we kept observing how things had changed. “When are we going to reach our goal?” we thought. It turned out that you never reach it. Your culture has to come to terms with the fact that there is no process goal. It’s about change and that change is continual.

Take for example a studio culture that was ruled by technology. Games were seen as platforms to demonstrate technical achievement. Tools and pipelines to improve productivity for artists and designers were always lower on the list of priorities because the programmers were always trying to dig their way out of the holes they created trying to accomplish impossible challenges. The build was rarely working due to all the bugs left along the way.

How did Scrum and agile change this? First it made broken things visible. They needed to have a working, and even potentially shippable, version of the game every two to three weeks. This raised a lot of problems. At first the teams spent half their sprints trying to cobble together a working build. This killed their velocity. Since velocity is only measured by the value of the features working in the game.

This simple measurement of velocity is important. Scrum relies on a simple psychological fact; people behave based on how they feel they are being measured. Scrum measures based on the value to the product added by a team. Teams make small changes to improve velocity. Our example team did this. They started to come up with ways to improve the reliability of the build. Because velocity is based on what is seen in the game which includes art and design, they started to form cross-discipline teams to improve their velocity. Because of the overhead of communication when the team sits apart, they moved together into the same area. Because the programmers began to see how the poor tools and pipelines impacted members of their team, they focused more on improving those things.

All of these changes occurred within teams that took ownership of their work. The role of management during all this was to facilitate the change. This is the hard part. This requires management to step back where they traditionally step in. A manager needs to become like the coach on a football team. They can give feedback to the players, but once those players are out on the field, it’s entirely up to them on how the team performs.

Does every team adopting Scrum see this kind of benefit? Absolutely not. Scrum is a tool for the team to build change their process and possibly their culture. It's like owning a hammer; it won't guarantee you will build a great house, but it's a nice tool to use when building it.

Friday, December 05, 2008

We are living in exponential times

If you think the exponential growth of game development budgets is unique, watch this:

Thursday, December 04, 2008

Fill your ipod with game development advice

Ryan Wiancko at Industry Broadcast has been recording podcasts of many game developer blogs and articles. Recently he has added a few of mine as well. Ryan does a great job at capturing a wide range of good posts from all over.

Wednesday, December 03, 2008

Who cares if it was called Scrum or not?

In distant past projects I encountered a strange phenomenon during the beta phase. Our team would enter alpha, the publisher would throw a couple dozen testers on the project and all of us would focus on the bug database. During this time I felt more productive than any other time of the project.

We would address the bugs daily. Weekly we would triage the bug database to prioritize the bug “backlog”. We tracked the total bug count and used a “burndown chart” to measure bug resolution velocity, bug discovery velocity and the projected “zero bug” date that we were all trying to reach. All I had to focus on was solving those bugs and achieving the best possible velocity.

Does this sound familiar? It’s no coincidence that many of the Scrum practices reflect theses practices. When people have a clear idea of what they need to do (in the form of smaller solvable tasks that they take ownership of), what the goal is and a solid empirical measurement that is updated daily, they can achieve a high level of focus and effectiveness.

It was one of those “aha!” moments when I came across Scrum and discovered how similar it’s practices were to what worked elsewhere.

Sunday, November 30, 2008

Team co-location and team rooms

Agile principles emphasize face-to-face communication wherever possible. The benefits of this are demonstrated best at the team level. When teams sit apart from one another, the overhead of communication and the problems that arise from a lack of easy communication are seen daily. Studies have shown that when teams reduce their physical distance between themselves, their performance increases in many ways(*).

Eventually teams realize this and rearrange their seating arrangements to improve communication.

Teams that co-locate are often limited in their options for team rooms. Often the office is arranged in small cubicles that cannot easily be removed due to power and data wiring limitations. Fortunately team rooms or “bullpens” are less expensive to create than cubicle farms or separate office spaces, so if you are lucky enough to influence the initial office space build out, you can create an ideal space for a team. Otherwise you’ll need to slowly adapt your current space. Some studios set aside a modest budget to slowly convert their spaces.

The Space

What makes an ideal team space? There is no ideal space for all teams. Sometimes teams can’t even agree among themselves what makes the best space. If you have a cross-discipline team, perhaps the programmers might want some windows while the artists don’t want the light from the outside. It’s up to the team to work out what is best. Here are some things to consider when deciding about your team room:
  • Is there enough wall space for your task board, information radiators and white boards? You can never have enough space for white-boards!
  • Is there enough “slack” in the space so that people can occasionally pair up at a computer? For example, can a programmer sit with an artist to discuss a problem?
  • Is there space for meetings, such as the daily stand-up or a play-through? Is there a dev station available to conduct a team play-through on?
  • Is the room “off the beaten path”? A lot of traffic from outside the team in the area can be disruptive.
  • Are there local private areas? More about these below.
Once you build your space, what kind of furniture is best. My opinion is that mobile, modular and adjustable furniture like those built by Anthro are best. Teams change and improve over time. The ability to rearrange the space a bit every Sprint can be a big benefit.

Concerns
There are typical concerns that are raised about team areas. These are:

“Programmers (artists, designers, etc) need to work in quiet isolation in order to focus and be effective. We can’t do this in a noisy team room with constant interruptions”.

This is a common concern about working in close quarters with other developers. For certain, there are more interruptions in a team room. Most of them are the point of the team room. When individual developers are trying to achieve individual goals, then interruptions often impede them from that goal. However agile emphasizes the delivery of value integrated into the game, not the value of finishing “parts” that should join seamlessly because a document says they will.

When a team is focused on achieving a common goal that requires everyone’s participation, fast access to other team members who can help progress is not only necessary but essential. It doesn’t make sense to have an animation programmer on the other side of the building implementing a system that may or may not be of any use to the animators someday. They should be solving the problem together.

Team rooms for people working on separate goals don't makes sense.

As far as noise level, if it starts getting in the way of progress, the team must solve it. Every team that I have seen transition to a team room doesn’t want to go back to isolation. If they felt isolation would benefit their performance, they would have permission to do that too.

This doesn’t mean that the people never need privacy. Having a few small offices around the team area where people can go to work on a tricky problem or have a private conversation is important.

“If I am not sitting in a group of people of the same discipline, then I won’t learn as much and be able to help them out as much.”

This argument is a more difficult barrier for teams who wish to co-locate. It has some validity. One example of this was in a studio where half a dozen AI programmers spread out across three teams. Following this, three incompatible AI solutions emerged that duplicated effort. However the best solution wasn’t to let them sit together in a single team. The reason was that they didn’t create the best AI system for the team. In fact they produced an AI system that was state of the art. It was rapidly developed and did amazing things in their test environment. Unfortunately this system didn’t meet the needs of the game. When matched up with motion captured animation in a representative level (rather than the test environment with ideal test shapes) the AI was incapable of even simple navigation.

What is the reason for this? Did the animators and level artists create the wrong assets for the AI or did the AI programmers create behaviors that were incorrect for the animation and level layout? It was really both. They should have developed everything together. They would have learned along the way to alter the behaviors and assets to accommodate reality. They would have collaboratively built knowledge of the best solution.

This drives cross-discipline teams and raises the value of instant face-to-face communication. What about the original problem of collaboration between members of the same discipline? How do we avoid the AI programmers going off in separate directions? One solution is orthogonal teams. Nothing prevents the AI programmers from gathering once a week or once a Sprint to discuss their work. It just doesn’t need to be as frequent as communication within a cross-discipline team.

* The effects of R&D team co-location on communication patterns among R&D, marketing, and manufacturing, Christophe Van Den Bulte, Rudy K. Moenaert, 1998

Tuesday, November 11, 2008

Beyond Scrum: Lean and Kanban for Game Developers

Gamasutra has posted an article of mine called "Beyond Scrum: Lean and Kanban for Game Developers". The article culminates what has been described here about using lean practices since 2006 and applied to games in production and details how Scrum teams can adopt Lean practices.

If you want to read more about Lean and reducing production costs by more than half:

Lean Production, first steps, September 2006
Google Talk with Mary Poppendieck, December 2006
Prototypes, April 2007
Book review: Implementing Lean Software Development, September 2007
Cooper product design applied to game development, parts one and two, August 2008
Phases in an agile game project, September 2008

Tuesday, November 04, 2008

Scrum and project miracles

There is a lot hype about the benefits of Scrum. Much of this hype is born out of the desperation developers have had from the failures using traditional methodologies. One example of this was on a project I joined in its early planning phase. We were challenged to develop a sequel to a game that another developer created. The original developer had spent four years developing the original game. We were given eighteen months to finish the sequel which had be of higher quality. The impression was that if we had the source code of the original game, we could be three times more efficient.

Detailed plans and schedules were created that defined how we would successfully ship a superior sequel on time. Unfortunately we failed to achieve the goal even after months of mind-numbing overtime and the project was canceled.

Could we have succeeded with Scrum? I'm sure we wouldn’t have shipped the game on time, budget and at the quality target. However I believe the project would not have failed as badly. Rather than trying to follow a preset plan and hoping to catch up as we slipped, we could have known earlier that the goals were impossible. Had we known early that the goals were unlikely, we could make better decisions on how to proceed.

The best way to spend less money on a game is to spend less time. We are always in a rush to be first to market. Even if costs are the same, we often choose the project with higher staff levels that can be shipped six months earlier.

This is why we are never given enough time to do our job well. This is why we are forced to enter production before we have found what is fun enough to produce. This is why we ship prototypes.

Scrum isn’t an answer to this. It’s a different way to deal with uncertainty by addressing it head on. Take for example, a game in development using Scrum that is projected to enter production by October 2009. After a few iterations, (see figure) we see that the actual progress of the team (the thick red line) is 70% of what was projected in the blue line as measured by user story points. By projecting this actual velocity (the green line) we see that with the current scope planned, we will enter production five months late.

The Product Owner (PO) will be aware of this actual velocity and has a number of choices to make. If the game really must enter production by October 2009, then they will likely cut scope. The PO could try to increase the velocity of the team by adding people, but this rarely works.

Conversely, if the PO wants all of the scope defined in the game, then they must communicate to the stakeholders that the production date needs to slip.

One caveat to mention here is that things aren’t often so cut and dry on our projections. We don’t often estimate user story points beyond six months in advance in detail and velocity isn’t constant. Beyond six months the crystal ball is a bit foggy. Some projects will estimate larger epics. Others will build larger backlogs. Neither of these solutions will give us more certainty. If the ship date is fixed more than six months off then we need to make sure the backlog is always prioritized so that if the PO needs to cut scope, they are cutting the lowest value scope.

The principles still apply. We need to measure the actual velocity and not depend solely on our predictions.

Tuesday, October 28, 2008

Discount for Scrum Master Class

I've received some feedback from readers that my Certified Scrum Master class for video game developers after the Montreal Game Summit November 20-21 is a bit too expensive for people who are not members of the Alliance numérique.

I agree and after speaking with the Alliance, they are allowing me to refer attendees to the course at the Alliance numérique membership rate. If you would like to attend at this price, please send me your name, phone number and email address where you can be reached.

Wednesday, October 22, 2008

Reminder: Certified Scrum Master for Video Game Developers course in Montreal

Just wanted to send out a reminder of the course I'll be giving after the Montreal Game Summit. The two day course will take place on November 20-21.

The last one sold out and we're half-way there on this one. Details at:
http://www.alliancenumerique.com/en-ca/agenda/20081101.html

Deadline for enrollment is November 1st!

Tuesday, October 21, 2008

Scrum is not for everyone

Some adoptions of Scrum won’t be too effective or simply fail at the studio or individual level. Conditions at a studio might not be at the right place to adopt Scrum. Some teams won’t take ownership of their work. Some leaders want to micromanage within a Sprint. Some team members refuse to participate in any team activity (such as a daily Scrum).

Many times adoption failure happens because studios don’t want to face the cost that fundamentally changing their culture might be. For example, there is a level of employee turnover that can occur adopting Scrum. Studios can expect to lose an average of 5% of their people when they adopt Scrum. More if they have a strong “command and control” culture. Some of these people will be valuable. They will leave because they don’t believe in the Scrum vision.

There are many reasons why people might leave or be let go. Let’s take a look at one common occurance.

Occasionally, the teams will “self organize people off the team”. This occurs between Sprints when teams are allowed to change their membership. Poor performers or poor team players can be “voted off the island” by the rest of the team. Some may view this as a potential HR nightmare, but it is necessary to allow teams to truly take ownership and make a commitment to their work. It can also be valuable for the person being ejected from the team. In most cases I have seen, this event is a true “wake up call” for the person told to leave the team. Usually this person has ignored years of management feedback about their poor performance or teamwork. However being ejected from a team cannot be ignored. It is a strong statement from a group of your peers.

At this point, another team has to be found that is willing to have this person join. They have to be made aware of the issues that led to their ejection from the last team. Teams cannot be forced to accept people they don’t want. The reason for this is you can’t expect a team to take full ownership and commitment in their work if they don’t have control over their own membership.

45% of time this person corrects the issues and becomes a valuable member of another team. Another 45% of the time, they find a team where the chemistry is better (you can’t ignore the fact that team chemistry is important). The remaining 10% of the time they move to another team which then has to eventually ask them to leave again. After a few ejections, it’s common to find that no other team will accept this person on their team. It becomes management’s duty to let this person go from the company. Sometimes the person gets the message and leaves on their own accord before this happens.

I’ve only seen this happen a few times. It’s unfortunate, but it makes a statement to the teams; they control their destiny. They are responsible and have the necessary authority to make changes to improve their performance. Failing to create these conditions for a team can lead to valid excuses not to achieve their potential. “We’re stuck with so-and-so and have to live with it” is a common refrain by teams that can’t self organize. They feel helpless to effect the change they need and therefore can’t achieve a high level of commitment.

When you see the performance of teams that achieve their potential you stop questioning the value of these practices. It does take a leap of faith to allow these practices into your studio. Sadly, some don’t take this leap and don’t realize the potential of hyper-productive teams.

Friday, October 10, 2008

Agile is not a "fad"

I've long become used to agile being referred to as "the next new management fad" or something similar. I don't agree with that assessment. When you focus on project management for a good portion of your working hours, you become familiar with the history of project management methodologies. You start to see it as a continuous evolution in the theory and understanding of how people work together. Agile practices have their roots that go back before the second world war.

However, when I hear someone who is credentialed by the "Project Management Institute" say this it makes me wonder. Don't they teach about the last century of management change? All practices have their roots somewhere. Current practices are just a snapshot in time.

The History

Before the 20th century, products were largely created by craftspeople. Things like carriages were made by hand, assembled out of parts that were not too interchangeable. This all started to change in the early 20th century with the assembly line and "Scientific management" (or "Taylorism"). I wrote about this a month ago:

Back in the early days of mass-production, when companies like Ford were discovering the efficiencies of assembly lines, the theories of Fred Taylor were being applied in factories. These theories were based on the idea that complex jobs, such as assembling a Model-T car, could be performed by relatively unskilled and cheap labor if each step in the assembly line was broken down to a simple job that could be standardized. Workers were considered to be interchangeable cogs in a large machine.

Though “Taylorism” has been discredited, many management practices are still based on the idea that as long as a well defined process and set of tools exist, then the work that can be done by a group of people can be predicted and defined up front. Workers are considered “fungible” or replaceable and anonymous on a large project plan that attempts to predict what each part of the whole will take a worker to complete. Managers find comfort in this. As long as people and hours are are interchangeable, then we can play with the amounts of each to achieve predictability.

Taylorism had been largely discredited before and during World War 2 by people like W Edwards Deming who taught that project management needs to facilitate workers to bring their thinking and passion to their job and focus on the whole system, not just their one part. Deming proved his work in the US during the wartime manufacturing expansion. Unfortunately many of these principles were abandoned in the US after the war. Deming found himself in high demand in Japan however and spent time there teaching these principles to Japanese industry after the war.

For the next three decades, much of the innovation in project management came from Japan and the results show the proof. Agile has it's roots in the practices developed at Toyota, Honda, Canon and other successful Japanese companies. Western companies such as Boeing, Ford and Volvo are still catching up with many of these practices (such as Lean production methodologies).

Software development in the west has been at the leading edge of applying these lessons to their products. One of the roots of Scrum is from a paper written by two Japanese business researchers in 1986. XP was built on Scrum practices as well.

Methodology dejour

Sure, there are a numbers of flavors of agile out there. Each week it seems as though someone else is branding their own methodology or misusing the term agile. Some of these are even more fleeting than fads.

In many cases some managers call agile a fad as a way to brush off the challenges of changing their mindset about how people best work together and what their role is in a knowledge focused organization. Perhaps they still think it would be better to manage 100 creative people through a Microsoft Project Server database from their desk.

Alternatively, don't just drink the Kool-aid and bet your company on agile. Step 1 is to learn more. This book is a great start.

Wednesday, October 01, 2008

The role of tools in agile planning

The value of tools
I had a great discussion this morning with the CEO of a company which makes a production management tool that supports agile methods. We discussed the issues of tool use on agile teams and the opportunities for providing tools that don't yet exist.

I am not a big fan of tools that help automate sprint backlog management. Management of the sprint backlog (tasks that the team estimates and updates to achieve their sprint goals) is best left to the team. Any tool that takes any of that ownership away from them is going to harm the team's ability to commit to their sprint goals and take control of improving their effectiveness. Sure, it might save the Scrum Master five minutes a day in producing a burn down chart, but the Scrum Master's role is to facilitate the team to become self managing and commitment driven. Sprints are about adding value and commitment, not about perfect task tracking.

On the other hand, tools for backlog management are very valuable. Product backlogs are living complex plans for the product and must be frequently updated by the product owner to remain relevant. Tools can help.

So what do we look for with tools?'
First and foremost, a tool used to support an agile team must be built with the agile manifesto in mind. The first value applies in full:
Individuals and interactions over processes and tools.

While we value processes and tools, we value individuals and interactions more. Are these at odds? No. Processes and tools can support individuals and interactions. Certainly any process built using Scrum properly supports individuals and interactions. What can tools do to support them?

Let's take mind mapping tools for example. Teams have built and managed entire product backlogs using these tools. Mind maps allow a hierarchical structure to be built and viewesdwithout losing sight of the whole. Teams can form around entire branches of features (e.g. online multiplayer team). Some of these tools allow nodes in the map (user stories) to be prioritized and estimated. During release planning meetings, having the mind map displayed on a projector or large printed format can focus the meeting. It contributes to collaborative meetings where everyone contributes. If you want to have some fun brainstorming a backlog, provide the team with a very large sheet of paper and half a dozen colored Sharpies and have them create a mind map right there.

Compare using Excel spreadsheets to manage the product backlog or release planning, which is common. Hierarchies are not clearly seen in the column-row format of a spreadsheet. It's easy to get lost in the details and forget the big picture. Although I love using Excel for numerous functions, when someone displays a product backlog spreadsheet on the wall, the room usually becomes quiet.

Mike Cohn has suggested another way to visualize a large product backlog. I'd love to have a tool that supports this view!

So how can tools contribute to the value of "individuals and interactions"?
  • Mix global and local views together. Have a view of the data (backlog, plan, etc) that can be zoomed in and out rather than page-flipped. Page flipping a view probably sheds 50% of the context for people in the meeting unless they never blink.
  • Focus on tools that capture collaboratively driven knowledge rather than isolated data entry. For example, have a tool that projects the release plan, one story at a time, and prompts for the story point estimates during a poker planning session. Don't create a poker planning tool that allows everyone to estimate from their desk. The design discussion during poker planning is the point of the exercise!
  • Information radiators are solid gold (burndown charts, unit test coverage, build stability, release plan, etc). Tools should print them out whenever possible and support large format printers.
Distributed teams can't avoid some of the problems with their reliance on tools. The fact is that distributed teams have a harder time achieving the same velocity as co-located teams.

The good news is that the tool makers are listening. New versions of tools are being developed (some even using the agile process) to better support the needs of agile teams.


Tuesday, September 30, 2008

Announcement: Certified Scrum Master for Video Games Course in Montreal, November 20-21

I'll be teaching a CSM course aimed at video game developers in Montreal on November 20-21. Here was the announcement sent out today:

Attend a workshop at the Montreal International Game Summit proposed by Clinton Keith, one of the speakers, and obtain a ScrumMaster for Video Game Development certification. The workshop will be given in English. But hurry; there are only 50 seats available!

Inscriptions are reserved to members until October 3rd. Afterwards; it will be opened to all until October 15. For more information (location, prices, etc.), click here.

Alain Lachapelle
Director, Montreal International Game Summit
alachapelle@alliancenumerique.com
T : 514 848-7177, 224

Friday, September 26, 2008

Phases in an agile game project?

In most agile projects outside the game industry there are no phases of development. Projects start with releases and every release delivers a version of the product to customers. Think of applications like Firefox. There is a new release of the browser on a regular basis.

Eliminating phases is a big benefit of agile; phases such as “a testing phase” force the critical activity of testing to be postponed to the end of the project where fixing bugs is the most costly. “Planning phases” at the start of projects attempt to create detailed knowledge about what features will be fun and the work associated in creating them. Unfortunately the best knowledge comes from execution which is why highly detailed pre-planning fails.

For many games however, there is often still a need to have phases within the game. There are two major reasons for this:
  • There is a minimum bar to the content being delivered regardless of the quality. $60 games must deliver 8 to 12 hours of gameplay. This represents the major portion of the cost of development and occurs after the gameplay mechanic is discovered.
  • Publishers have a portfolio driven market model. This constrains the goals of the games that they fund. In order to gain publisher approval (which includes marketing and often franchise/IP owner approval), developers need to create a detailed concept treatment at the start of a project.
The first reason compels us to separate pre-production and production activities into separate phases. Pre-production allows more freedom to iterate on ideas and explore possibilities. During production, we are creating thousands of assets that depend on what we have discovered during pre-production. These assets create a cost barrier to change. For example, consider a team in production on a platformer genre game. Platformer games challenge the player to develop skills to navigate treacherous environments (such as Nintendo’s Mario series). The team will create hundreds of assets that depend on character movement metrics such as “how high the character can jump” or “the minimum height that the player can crawl under”. Production assets depend on these metrics.

If these metrics are changed in the midst of production, it can wreak havoc. For example, if a designer changes the jump height of the character, hundreds of ledges or barriers would have to be changed. This can create a great deal of wasted effort during the most expensive phase of development. It’s critical to discover and lock those metrics during pre-production.

This doesn’t mean that we can’t be agile during production. How we are agile does change. Instead of using an iterative and incremental process such as scrum, a more incremental process such as Lean is more applicable. More later on Lean....

Wednesday, September 24, 2008

Black Rock Tricks Out, Gets Agile With Pure

Interesting article in Gamasutra. Jason Avent talks about their experiences with agile:

And they define how they do the work, and what work they do, and what order. That means that because they're closest to the work, closest to the jobs that need doing, they make better decisions than the people who are further away.

So it makes people happier, it makes people much more productive, it means they don't have to work such long hours, and the product is better. It's such a hands-off management method, it does make you bite your fingernails a bit, initially, but once you see the results, it's huge.

----
It can take a bit more bravery than "biting your fingernails a bit". Giving autonomy to teams, even for a couple weeks at a time, is too much for some managers.

Another cause of adoption failure is hiding from transparency; Scrum, done correctly, exposes everything that is wrong with your development environment. Some managers will see this flood of exposed flaws as being caused by Scrum. Instead of addressing the actual problems, they will stop the experiment with Scrum and everything goes quiet again. Problem solved!

Whenever I visit a studio to talk about agile and Scrum, there is always that person or small group of people that are at the center of the adoption. They are taking a chance introducing and championing a new way to think about how people work together. I'm always impressed by them.

Tuesday, September 23, 2008

Early Consumer Testing

There is a good article on Gamasutra about the merits and precautions of early consumer testing from the experiences of NetDevil. More developers are using Scrum and have builds that show value very early in development. This creates an opportunity to test with consumers early to inspect what is resonating with your potential customers much sooner.

NetDevil is experienced Scrum developer in Colorado. They have fully committed to adopting agile and sent several people to the first Certified Scrum Master for Video Games course in Austin four months ago. In the article Ryan Seabury, the lead producer for Lego Universe, cautions about taking things too far with reacting to what the consumers report. This reflects High Moon's experience with early testing as well.

At best consumer testing is a gut check for the developers and especially the Product Owner on the direction being taken with the game. It also has another great benefit; When the team knows that their build is going to consumer testing, their attention to detail and polish is raised. The team looks at the build wearing their "consumer goggles".

Monday, September 22, 2008

Shared Infrastructure Teams

Shared Infrastructure (SI) teams provide low level support such as engine, audio, online, etc services that multiple games rely on.

A frequently asked question is how shared infrastructure (SI) teams should organize themselves in an agile project environment. Since they support multiple teams, they receive requests for features that cannot easily prioritized as they are for a single team. This can create confusion and conflict between the SI team and their "customers", the games that depend upon them.

I've found that a number of practices are valuable for these teams:
  • SI teams require their own backlog and product owner (PO). Having more than one backlog and one PO is a recipe for disaster. The team should have every benefit that other agile teams have in an understandable backlog and single vision.
  • Customer teams should identify priorities during release planning and include the SI team (or at least their lead and PO) in their release planning. SI teams usually need a longer planning horizon than a single sprint.
  • SI teams should factor support into their velocity whether it is identified for tasks or not. Setting aside a certain percentage of your bandwidth for unexpected maintenance is critical.
  • Loaning SI team members out for a sprint is OK, but it should be identified in customer release planning. It's very valuable to have SI team members see how their "product" is being used.
  • The SI PO should ideally be at the executive level (or in frequent contact with them) to arbitrate conflicting product priorities. Deciding to support one game over the other is a company level (strategic) decision and should have the input from the people that run the studio. For example, the CTO should be the PO for the SI team (how's that for an acronym loaded sentence?).
With their own PO and backlog, an SI team can feel like a real team and take ownership of their work.

Monday, September 15, 2008

A Better Planning Method

I'll never forget the day I experienced my first "publisher flame-thrower phone call". At the time I was the Director of Product Development at Angel Studios. We were six months away from the ship date of Midnight Club 1, a PS2 launch title. I called an executive at Rockstar to tell him that we needed to drop one of the three cities we were planning to ship for the game. I can't describe the phone call other than asking you to imagine calling Leonidas (from the movie 300) to tell him you had just sacked Sparta behind his back.

I really didn't expect such a strong reaction. We had originally written the design document to include six cities. We had cut that down to three as we discovered how difficult it was to create full cities for the PS2. During the first phone call, they accepted the cut calmly, I later learned that the final cut from three cities to two had come one week after a large marketing blitz which told the world that Midnight Club would have three cities. Had I called them a couple of weeks earlier, it would have been a much different call. Of course, had we put two cities in the original design document, we would have been heroes by shipping on time and budget AND with the original scope. As it was we shipped on schedule and budget with one third of the scope we had "promised".

The uncertainty of how accurately we can plan decreases with how far we plan ahead. We often get into trouble by relying on our plans too much and ignoring reality. We base complex and highly interdependent schedules on a plan we assume is comprehensive; a plan that anticipates every potential risk. We then discover that reality doesn’t follow our plan and the amount of work we need to spend is far greater than our budget allowed for.

One reaction to this problem is to ask “why plan at all?”. Indeed, many projects have launched with little planning. A project without a plan or schedule can be appealing at first glance, but there are problems this raises. Maintaining a vision becomes a problem, especially with larger teams. Strong leadership can help overcome this, but that leadership will often become a bottleneck. Developers of sequels to smash hit games will often announce that the game “will be done when it’s done”. Unfortunately even this formula doesn't always insure that the success of the sequel will match that of its predecessor.

There are often schedules outside the team that need to be coordinated with key project deliveries. Publishers often have marketing budgets that drive the rate at which projects can be released. Portfolios of games are balanced around key selling seasons or movie co-release dates. Few developers have the luxury or of ignoring these pressures.

The reality is that most hit games released have missed their original ship date, budget and scope. Detailed planning, bloated budgets, staffing and crunch inflicted on the developers haven’t proven to be a cure.

So what value does planning have? It isn’t so much about creating an accurate schedule, cost and feature set for a project up front. It’s an ongoing quest to refine those values over time through iteration. It's used by the team and customers to balance schedule, cost and features. Any planning method has to acknowledge the uncertainty of these three elements of a project up front and focus on refining the plan continuously by doing the following:

  • Reducing risk - You can’t plan away uncertainty. A plan should acknowledge risk first and foremost. Addressing risk requires visibility and transparency.
  • Creating knowledge - A “Big Design Up Front” (BDUF) usually fails because we don’t know enough to make the decisions that we make up front. A planning process has to build on what we learn through iteration.
  • Communicating information - A good planning method has to communicate changing information properly. BDUFs fail at this because teams and customers don’t reread the document for changes. Frequent effective meetings between the team and stakeholders to update the plan are essential. Instead of spending months planning up front a good planning method spreads the planning time across the project.
  • Supporting better decision making - Too often bad games are released because the decision to cancel it came too late in the schedule. Publishers simply hope to recoup some of its budget through selling the game to a hundred thousand unsuspecting players. A proper planning process would allow better decisions to be made earlier which would steer the game to profitability or cancellation before too much money is spent.
  • Reflecting realistic progress - We would prefer to know early in the project whether our plan is realistic or not. Say we have to ship our game 12 months from now and our current feature set requires 18 months based on our current progress. Knowing this information can help us pick and choose a smaller set of the most valuable features or decide that we have to slip the release date. Conversely if we realize we are going to miss the ship date 3-6 months before we are scheduled to slip, we may not have the same range of choices: We might be in the middle of production on a fixed set of mechanics. A good planning process should have a feedback mechanism built in to reflect reality. It should build trust between the publisher and team.

The goal of agile planning provides all these areas of support. If you want to find out more, buy this book.

At the very least, you can avoid a phone call like the one I had!

Tuesday, September 09, 2008

Tools for building the product backlog

I’m opposed to using tools to automate the daily scrum. These tools detract from full team collaboration that needs to take place to be successful with Scrum. However there are strong benefits to using tools to help facilitate the gathering of user stories into a product backlog. Epics and stories are not a scattered collection of ideas for a game. They form a hierarchy of requirements that are disaggregated and updated throughout the life of the project.


Many tools can store and maintain a hierarchy of data. A simple database can be constructed to do what you need quickly and inexpensively. However, there are a number of features to look for when choosing or even designing a tool for this purpose:
  • Graphical display - It’s best to display the hierarchy on a projector during a story gathering workshop. This shows the big picture of the product backlog and where new stories are being added in the hierarchy.
  • Dynamic editing and display of branches - Sometimes entire branches of the tree will be moved or deleted during the workshops. It’s best if this can be done by right-clicking or dragging the branch you wish to change. Sometimes the group will want to focus on a single branch in detail. It’s very useful to be able to collapse all the other branches and just display the branch the group wants to discuss.
  • Graphical options for individual stories - Stories may be prioritized, flagged for attention or have additional information attached to them during the meeting. A tool that is extensible and allows meta-data attached is beneficial.
  • Flexible - The tool must not impose too much structure. It should allow the creation of sections such as a “parking lot” for future story ideas, etc.
  • Powerful export capabilities - The tool should be able to export data to a variety of popular and readable formats for sharing with customers who do not wish to buy or learn the tool. Export formats like Word or Excel are a must.
The one thing to be careful of with any tool is that the person with the mouse who is making the changes does not take charge of the meeting. Don’t let the product owner near the laptop in the meeting. If the person with the mouse starts to filter everything they hear, then their voice will naturally dominate the discussion. The mouse gives them the illusion of control. It will dampen the contribution of everyone else at the meeting.

I have used mind map tools such as MindManager and FreeMind for building and tracking product backlogs in the past and can recommend them.

Friday, September 05, 2008

Player roles and user stories

Many game development projects don’t put much thought into the various kinds of players who buy the game. They usually add three levels of difficulty towards the end of development as a means of adding replay-ability and accommodating a range of player skills. The levels are differentiated by a simple scaling of challenges in the game such as the number of opponents, the damage from their hits or the damage your hits cause. This reflects the amount of effort we think it’s worth.

Would we benefit from considering a broader range of players and placing more importance in their roles throughout development? Some games do this, especially some online games. An example is the popular Battlefield games which allow players to equip themselves based on specialties. If you are not familiar with the games or the specialties, they are usually divided across these roles:

  • Assault specialist - Equipped with an assault weapon and grenades for close quarter combat.
  • Sniper - Carries a high power sniper rifle and a sight that can they can use to call in precision strikes.
  • Engineer - Has a bazooka, mines and can repair vehicles.
  • Special forces - Carries a light automatic weapon and C4 explosives for sneaking around behind enemy lines causing problems.
  • Support - Totes a heavy automatic weapon and a radio to call in mortar strikes.
These specialties require different behavior from each player who assumes each role. They aren’t as limited as difficulty levels because players can try each specialty in any order and with any skill level. These specialties cannot be added at the end of development. They need to be developed somewhat in parallel during preproduction. They have an impact on level design and should be added well before production starts.

User stories provide a mechanism for identifying these roles and clearly communicating features related to each. The template for user stories I like is:

“As a <>, I want <> [so that ].”

A good method for identifying and differentiating goals is to phrase the user story in terms of those roles. So instead of saying:

"As a player, I would like to have a bazooka so I can blow up tanks"

the story becomes:

"As an engineer, I would like to have a bazooka so I can blow up tanks".

What's the difference? It's mainly one of value and priorities. For a generic player, the bazooka is one of a host of weapons, many of which are more important to the game. However for the engineer, the bazooka is probably the most valuable weapon. I wouldn't play the engineer without it. There's nothing more gratifying than taking out a tank with a well placed shot.

Even if your game isn't going to have specialties, like Battlefield, there is a lot of value in brainstorming the various roles of players early in development. Who is buying your game? Are you going after a largely casual market? If you are, it would benefit you to identify the "casual player" role in some of your stories. It will lead to many small decisions such as simplifying the controls or adding more checkpoints so the casual gamer doesn't become frustrated.

Tuesday, September 02, 2008

How Pixar Fosters Collective Creativity

HBR is posting the article free for a short period of time.

What's equally tough, of course, is getting talented people to work effectively with one another. That takes trust and respect, which we as managers can't mandate; they must be earned over time. What we can do is construct an environment that nurtures trusting and respectful relationships and unleashes everyone's creativity. If we get that right, the result is a vibrant community where talented people are loyal to one another and their collective work, everyone feels that they are part of something extraordinary, and their passion and accomplishments make the community a magnet for talented people coming out of schools or working at other places. I know what I'm describing is the antithesis of the free-agency practices that prevail in the movie industry, but that's the point: I believe that community matters.

Thanks to Clarke Ching for pointing to the article.

Overtime works with waterfall?

Many Scrum teams have found that excessive overtime reduces productivity. The frequent inspection of work done on a daily basis makes measuring productivity far easier with Scrum. One of the reasons for this is that a Scrum team can adapt their practices and see what effect those changes have on their effectiveness.

Jeff Sutherland reports that one of the companies he coaches has measured the productivity of teams using Scrum and waterfall-like practices under different overtime conditions. They produced a graph they call the Maxwell curve:


This is hardly a scientific study (e.g. I'm real curious about how they measure story point velocity in a waterfall environment) but it is a very strong visual argument for what is intuitive about how people work:

- Teams of people who take ownership of their work and make a commitment are more productive, but this high level of productivity cannot be sustained for 60 hours a week.

- When people are treated like cogs in a machine (handed estimated tasks that have to be completed to a predetermined schedule), they can indeed produce more at 60 hours a week that 40. However the productivity of cog teams is not nearly as high as committed teams because their intensity is not nearly at the same level.

Think of a runner sprinting and a jogger. The sprinter will be faster, but cannot maintain that pace as long as the jogger.

The question is "who covers the greater distance?". Does the team that "jogs" go farther in 60 hours than the team that "sprints" for 40? Maybe, but which is sustainable? Which team would you rather be on? Also, is it the same progress? Consider Jeff's comment on overtime with waterfall:

Overtime doesn't work in waterfall. It introduces technical debt. It works short term for the project leader as long as no one discovers he is damaging the code base. Velocity gets slower and slower with overtime but it may be years before management realizes they have to pay for the technical debt. By then the project leader has been promoted.

Monday, September 01, 2008

Jidoka, TDD and asset validation

Jidoka
Jidoka is a Japanese term used at Toyota which means "automation with a human touch."

It's origins lie in the early philosophy of Toyota (at the time it was called Toyoda). Part of this philosophy was to minimize labor costs by reducing labor waste. A example is the creation of "Type-G automated loom" in 1924. Before then, each loom was watched for thread breakage by a single operator. If a broken thread wasn't caught quickly, it would ruin an entire run of cloth. The entire process was very wasteful in operator time (90% waiting) and ruined cloth.

The innovation of the type-G loom is that it would automatically stop whenever the thread broke. This allowed a single operator to support dozens of machines and virtually eliminated bad production runs of cloth due to broken threads. Quality went up.

When Toyota started making cars, the philosophy of Jikoda was carried over to the manufacturing process. On the Toyota factory floor, a problem can potentially stop the entire line until it is fixed. Once the fix is identified, the standard process is improved to prevent a recurrence of it in the future.


TDD
Those of you using TDD (Test Driven Development) should recognize this flow. Unit tests are introduced for every function in the codebase. These unit tests provide validation of those functions that are run when changes are integrated into code base (by a continuous integration server).

When we discover a bug, we must solve it immediately or end up stopping the line (stopping the commits). When the bug is identified, a fix and a unit test, to catch further recurrences of that bug, are checked in and work continues.

Why solve bugs immediately?
  • Bugs can cause the entire team to lose work. A build that crashes can waste hours of work across the entire team.
  • Bugs cost the least to fix immediately after they are created. Bugs fixed months later in "alpha" can cost 10-100 times more.
The practices for TDD are well established. Tools like Cruise Control allow an easy integration of TDD into any development environment.

Asset Validation Jikoda-style

What we need are similar tools and practices for a version of asset TDD.

Unit testing for assets should:
  • Catch assets which break the build
  • Catch assets with naming convention errors
  • Catch assets which violate budgets (texel density, memory footprints, poly counts, bone counts, etc).
  • Identify and track approved assets versus unapproved assets that need art lead approval
Bad assets are another form of debt, like bugs. They cost more to fix later. A more automated approach to checking assets will help keep this debt, which is waste, low.


I've always been on teams that do some of these steps, but they weren't complete and they were always implemented later than they should have. It would be great to have some more standardized tools. Hey Autodesk!.....

Friday, August 29, 2008

Should the Scrum Master also be a member of the team?

This question is raised frequently. I prefer that a Scrum Master (SM) not be a member of the team, but this is not a strict rule of mine. An SM as team member can create problems:

  • A focus on their own tasks over the team issues.
  • Prioritizing their own impediments over those of other team mates.
  • It can impede team ownership through the “Scrum Master as Team Lead” anti-pattern.
These problems are usually created subconsciously, but they do create barriers to team commitment. Sometimes there is no choice but to have the SM be a member of the team. There may not be enough teams for the SM to fully occupy their time. An SM can handle 3-4 teams before it starts becoming a full time job (your mileage may vary).

If the SM has to be a member of the team, keep an eye out for these problems. Raise them in the retrospective. It’s not easy, but it has to be done. One great way to overcome this is to rotate SM duties. If your current SM is the only one certified or experienced, they should coach the other people who rotate into the role.

Rotating the SM role has many benefits. Team members appreciate and understand the role much better after trying it.

Wednesday, August 27, 2008

Survey - When and where to hold next CSM class?

I've received many requests for another 2 day "Certified Scrum Master for Video Game Developers" class (the one in Austin last May filled up).

So I'd like to collect a your opinions about when and where to hold the next class. If you are interested in attending one of these, please answer a few questions in the short survey below.

Click Here to take survey

Thanks!

Tuesday, August 26, 2008

Agile values - Individuals and interactions over processes and tools - Part 2

Question: How does a large software project get to be one year late? Answer: “One day at a time!” - Fred Brooks

Our processes and tools to manage ever growing projects have built up over decades. Large teams have driven the creation of hierarchies of management. Project schedules and design documents that attempt to predict every requirement and task necessary to make a fun game now require expensive databases to manage. All of these are considered necessary to tackle the complexity that rise from having upwards of a hundred people working on a project for years.

Game development requires developers of widely different disciplines. Take for example a cutting edge AI character that needs to walk around an environment and challenge the player in the game. The creation of this character requires the participation of animators, designers, character modelers, texture artists, programmers audio composers among others.

It’s important that these disciplines collaborate as much as possible to be the most effective. If the animator is experiencing a problem with the animation technology, then it is important for them to work with a programmer as quickly as possible to address the problem so they can continue working. Often the processes and organization will prevent that. In our example with the animation bug, the programmer may be working on a series of tasks that were assigned by a lead. This may prevent that programmer from helping out the animator without permission from their lead. This leads to a chain of conversation as shown in the following figure:



The animator has to pass the request up through the chain of command which then has to make it back down to a programmer who can solve the problem. In this example, the change involves five people and four requests! This flow is prone to failure and delay.

So what has been described?
  • Over 100 people of various disciplines on one team.
  • Thousands of unpredictable problems that can introduce wasted time and effort
  • Detailed plans and tools to manage them that can’t predict these problems and are potentially inflexible at changing.
  • Hierarchies of management that can lead to further waste.
Agile can address these issues from the bottom up. This is a major benefit from self-managing teams. They self-manage the smallest level of details, not necessarily the highest levels. They aren’t self leading teams. They unburden leadership of the role of managing minor details. They allow leadership to focus on the big picture.

When teams learn that they can take a small amount of ownership to solve the smallest problems, they start taking on larger problems. They begin to ask for more ownership in other areas:
  • In creating better teams that can solve more problems by reducing external dependencies.
  • In demanding more of themselves to achieve commitments they own.
  • By identifying risks early and addressing them before they become problems at all.
  • By growing leaders among their own ranks.
Agile values are preferences and not binary decisions. We can still have process and tools to support the team, but we value individuals and interactions more to solve most of the problems that occur on a daily basis.

Scrum Rising article posted

I've posted my February 2007 Game Developer Magazine article "Scrum Rising" here.

It's an overview of applying Scrum to game development.

Monday, August 25, 2008

Scrum makes you smarter

http://jeffsutherland.com/scrum/2008/05/scrum-makes-you-smarter.html

"Recent research shows that people who work in command and control environments actually lose a significant number of points on their IQ. They get stupider."

Sunday, August 24, 2008

Agile values - Responding to change over following a plan

“Plans are nothing. Planning is everything” - General Eisenhower.

When the allies stormed the beaches of Normandy in 1944, they had a very detailed plan of attack. This plan led to their assumed victory over Germany before Christmas of that year. That plan was doomed from the start. How could it be otherwise? How could a plan for hundreds of thousands of soldiers, millions of pounds of supplies and thousands of missions all work out according to a plan on paper? Eisenhower knew it wouldn’t. That’s why there were always contingencies. That’s why the allies were always adjusting the plan based on the reality of the emerging battles.

Agile project management follows similar reasoning. A popular misconception about agile is that it doesn’t allow for plans. This isn’t true. Agile focuses on the activity of planning rather than focusing on a fixed plan. A certain amount of knowledge about any project is going to be well known. Just as the allies knew the location of the beaches they were going to land on, a game development project will begin with quite a bit of knowledge about the genre and market targets for the game to be developed. It is necessary to plan for these things and to share that plan. The problem occurs when the planning goes too far. I’ve seen a game design document for a fantasy shooter game that had the bullet count for each of the clip types of every weapon. How can we know how many bullets per clip we should have in a game? Why do we need to plan for that detail before we have the knowledge of what we need? This is an example of the source of problems that detailed plans can create. If the team sticks to the detailed plan, then it won’t be the best game possible. Great game emerge.

The agile approach is to plan for what is known and to execute against what is not known iteratively. In other words, agile is focused on gaining knowledge. If we don’t know how many bullets in a clip or what type of weapons will be the most fun, we don’t try to plan away that uncertainty on paper. We address it head on by building weapons early and learning what is best.

Many elements of a plan are assumptions. We assume the things that we put into a game design document are going to contribute to a hit game. Some of these assumptions are correct, some are not. By implementing and testing these assumptions, we change some of them. We then alter the plan to create a better game.

Saturday, August 23, 2008

Looking for a great programmer who does TDD?

A former co-worker. Very talented and only interested in a US game developer who embraces agile & TDD. Highly recommended. Contact me.

Thursday, August 21, 2008

Iterations and vertical slices

Any game developed using agile makes progress using iterations. The goal of every team for every iteration should be to make progress by adding value to the game or pipeline for a customer. This might be a the improvement to a core feature for the player who buys the game or a function for the animator using the asset pipeline or a tool.

Iterations (or Sprints) are like mini projects by themselves. They often include design, coding, asset creation, tuning and debugging. However we are not always producing full vertical slices of a game every iteration. We’ll use a an example of what we might deliver for a team which is focused on creating AI behaviors: One of the most difficult aspects of AI behavior is navigation in a complex environment. The AI logic has to identify objects that will prevent the NPC from moving and calculate a path around them. Throw in some other moving characters and objects and the problem can become very complex to solve. Navigation can become the most riskiest problem to solve AI and therefore one the the riskiest problems to solve for the entire game.

Naturally we would want to solve the navigation problem as early as we could. The problem with doing this work early is that other related systems such as character animation and physics might not be developed enough to support the sprint goal of having a polished AI character walk through a complex environment. In this case, the iteration goal for the team might be to demonstrate simple cylinders navigating a complex test environment.

Does this solve all the risk associated with AI characters navigating complex environments? No. It probably addresses a big part of the problem, but there may be other problems that show up when progress is made with the animation and physics. The benefit is that those systems will have some basis to work against. If we find that our cylinders have problems navigating up stairs, then that might influence animation and physics work over the upcoming iterations.

It's a much more intractable problem when you discover your nicely animating AI character can’t climb the stairs that are duplicated in every level.

Iteration goals are prioritized based on value, cost, risk and knowledge. Addressing high risk and creating knowledge sometimes drives the highest priority stories on the backlog. A Product Owner has to understand the risks of the project as well as the value of the features when prioritizing.

Wednesday, August 20, 2008

Agile values - Individuals and interactions over processes and tools

The agile manifesto defines four values that the criteria for any method which calls itself "agile". I'm going to cover the ideas behind each:

Individuals and interactions over processes and tools

Back in the early days of mass-production, when companies like Ford were discovering the efficiencies of assembly lines, the theories of Fred Taylor were being applied in factories. These theories were based on the idea that complex jobs, such as assembling a Model-T car, could be performed by relatively unskilled and cheap labor if each step in the assembly line was broken down to a simple job that could be standardized. Workers were considered to be interchangeable cogs in a large machine.

Though “Taylorism” has been discredited, many management practices are still based on the idea that as long as a well defined process and set of tools exist, then the work that can be done by a group of people can be predicted and defined up front. Workers are considered “fungible” or replaceable and anonymous on a large project plan that attempts to predict what each part of the whole will take a worker to complete. Managers find comfort in this. As long as people and hours are are interchangeable, then we can play with the amounts of each to achieve predictability. This idea has been disproved for decades. It was the basis of Fred Brooke’s book “The Mythical Man Month” in the sixties. People and months are not interchangeable.

This is why agile values individuals and interactions over processes and tools. Teams of people make games. Artists, programmers, designers and testers all have to communicate to be the most effective. Collaboration and shared vision are key to great teams. Tools and process are still important, but they can’t replace this.

So why do we cling to this idea? Because it’s easier. Process and tools, interchangeable hours and people, spreadsheets that are meant to be crystal balls only require management. Collaboration and shared vision is harder. They requires leadership. Leadership is harder than management.

Tuesday, August 19, 2008

Production Debt

Have you ever seen a schedule for a game project that picks a day for production to begin? I think I've seen that on every project. Some day is chosen, a year or so in advance, and you have to start creating "production assets (levels usually)" on that day. I've never liked it and it's never worked out very well.

The problems are:
  1. Teams never transition from pre-production to production as if a switch were thrown. Different asset types transition at different rates. E.g. we may have level production down, but our animation system is still lagging.
  2. Where does this date come from? How do we know how much time we need for pre-production, and more importantly how do we know how much time we need for production, especially when we have a fixed ship date? How do we know if the time slot we're giving ourselves is enough so that we don't paint ourselves into a corner? How many times have you entered a nine month production phase with 12 months of work to do?
  3. What are the criteria for determining we CAN be producing assets? The date comes and we have to be in production, so we usually tell ourselves and our publishers "sure...we're in production! All those assets you see there are really production assets that aren't done...yeah". You're still iterating to find the fun and you can't iterate as much in production!
Measuring Production Debt in Preproduction

Production work is a necessary debt that we create in preproduction. One of the goals of preproduction releases (or milestones) is to measure that debt. During the first few releases that debt will be uncertain. You might say "we believe that we have 1000 people months of production work, plus or minus 200". The project would continuously refine that number during subsequent releases. Towards the end of preproduction you want it to be a much more accurate number.

Why Measure Production Debt?

Imagine you are on a first person shooter project that has 12 levels planned and 12 months of production scheduled. During development your team implements some cool technology that allows every piece of non ground geometry to be destroyed or have holes blown in it. This is a great new addition but it creates a problem. It doubles the amount of effort required to build a level.

How do we react to this? The first problem is that we have to know the problem exists. In many cases we don't do a proper job of measuring the production impact of design and technology changes. We paint ourselves into the proverbial corner. This is why demonstrating the cost of production should be a part of every release review.

If we know that we will double the level production debt with a new feature developed, such as the destructible geometry feature, we can make solid decisions early. Some choices would be:
  • Drop the feature because we can't afford it
  • Drop half the levels
  • Start production earlier
  • Extend production and ship date
  • Scale up production resources
Some of these choices are harder than others, but they are all better choices than the one that is usually taken: trying to stuff 24 months of production effort into 12 months of schedule.

How to Measure Production Debt

The only way to measure production debt is to make a small set of production quality assets, measure the effort and scale the measurement. For our example, our release would demonstrate a small level with a couple of buildings that are destructible. If we estimated that the demo level was 10% of an entire level we could say that the level asset debt would be 10 times the effort spent on the demo times the 12 levels we are planning.

As mentioned this would be an inaccurate measurement in early releases. We can't always make truly shipping quality demo levels early in a project. Subsequent releases would get closer and the estimate more accurate.

The good thing is that we can expect that we'll get more efficient at creating levels during production and that will free up time to incrementally improve the levels during that time.

What Happens to the Production Date?

Publishers will rarely allow a team to tell them "we'll pick a production date when we know more". It's OK to pick a production date with the caveat that the date will be refined per asset type every release. Back the refinement up with higher quality asset demos every release. Publishers and developers should have ongoing discussions of the questions raised above when it's clear that new features are impacting production debt. Publishers rarely get good information about cost and benefit information about major features.

Production debt (like every other debt it seems) always grows. The problem with production debt is that we can't retire it in an ongoing was as other debts (design, technical, etc.) are. Don't make your team attempt to pay it off with crunch.

Sunday, August 17, 2008

Cooper Product Design applied to Video Games - Part 2

In Part One, I described Alan Cooper's model on how agile fits in with his Goal Driven Design process. It made a lot of sense to me and it bears examining if it can be applied to game development.

As discussed, Cooper doesn't shy away from applying phases to product development...something that many agilists do. However his phases are more integrated and overlap their responsibilities throughout the entire project. So rather than calling them phases, he refers to them as "stages".

Comparing this to games, we typically have two distinct "stages" that be identified in game development: preproduction and production. I added the concept stage in part one since games do not start iterating from a blank slate on day one.

To better apply the Cooper design principles, I'd now like to divide up the pre-production stage into two stages: design and engineering. These are not phases divided by time. Both are intertwined, but they are answering two different questions. Design is focused on "what" we are building. Engineering is focused on "how" we are going to build it. This would include the tools as well as the engine. They are really both design stages, but the goals of what they are designing are a bit different, so I'll separate them here.

Here are the four stages we are going to talk about:


We ask questions of the first three stages and make a statement at the production stage:

The stages that ask questions seek correctness. They want the right answers. In the production phase, we seek to maximize efficiency while we are building. We know the answers from the previous stage. Entering production without these answers is one of the most common reasons for cost and schedule overrun.

The "toolsets" for each stage are different. In the concept stage we use our intuition to determine which path to take. In the design stage, we're focusing on player and stakeholder needs. The engineering stage focuses on how these ideas are going to be realized and if it's possible to produce with the resources we have . The production phase focuses on efficiency; how can be build things faster, better and more inexpensively.

There are three different states of mind that for the four stages. The two middle stages are where collaboration occurs among mixed discipline teams. The concept stage is more solitary. It's driven by vision and insight. Consider visionaries such as Miyamoto. Their major contribution comes from their vision and concept of game ideas. You can't take a group of non-Miyamoto (lesser) game designers and expect them to scale their vision to his level.

The production phase is not as collaborative either, especially in cases where we are trying to outsource production assets. It's more about the flow of productive work and improving the flow. This does require some collaboration, but it's not the same. We shouldn't stop to discuss and change the goals with the customer at this point. Changing genres when half your levels are produced is a bad thing.

Finally, let's look at the procedural approaches we use in each stage. In the concept stage, it's pure iteration. New ideas are tried and old ones are discarded. We have total freedom to explore in any direction necessary. Insights are not incremental.

The design and engineering stages are iterative and incremental. We increment towards production through time-boxed iterations that improve value toward customer value (humans) and the means to provide it (technology).

The production phases is incremental. We are not iterating on things we discovered in preproduction. We're mass producing the assets and finding ways to incrementally improve how we create assets.

Where does agile fit in?

The two center stages are the agile stages. This is where fully collaborative cross-discipline teamwork takes place to push the product forward and achieve the best results.



The other stages Cooper refers to as the "fragile stages". The concept stage is "magical" and unpredictable. It's hard to know when it is successful. You can't create a large team do it faster. The production state is fragile because of its size, length of time and potential for waste.



However with the production stage, we can apply Lean principles which are largely derived from product manufacturing (e.g. Toyota). These focus on eliminating waste and focusing on where value is added. It focused on a constant set of incremental improvements that do support collaborative work but also work standards for larger groups of people.



Conclusion

I feel this model has some value. Over the past five years, I've considered & discussed a couple of the issues with adopting agile for video game development:

- Game development is a combination of product development (preproduction) and manufacturing (production). Applying the same Scrum practices to both is not ideal.

- Our one release model for large scale games isn't an ideal fit with the regular release model of other products where agile is more commonly applied. We can't get proper customer feedback for our iterations. Focus/play testing is not enough. We need visionaries.

Stages seem a good fit to describe what we do on an agile game project. Cooper's breakdown adds value in discussing the practices, tools and goals of each stage. Stages also address the problems of phases by not handing off responsibility between them.

Scrum is a good fit for the design/engineering stages. Based on production experiences, Lean is an ideal process for the production stage. Many studios do the concept stage, but as Cooper says: "it's magical". How did Apple create the concept for the iPod? What practices do we create for the concept stage that allows many ideas to be inspired and filtered? How do we know when we are done?

Cooper's model is a step forward is defining that.