Incremental and Iterative development on a huge scale

An article in this morning's San Diego Union Tribune described the 10+ year, $160 billion dollar Future Combat System (FCS) project. The following section caught my eye:

... they are using an innovative development model, which they say is better suited to a huge project with multiple interlocking parts.

Zanini said FCS is being developed in phases. At each stage, products are tested by soldiers to see if they work and how they can be improved, he said. Lessons learned from testing are applied to the next phase of development, which allows the program to keep up with advances in technology, he said.

“You're in a build-test-build cycle throughout the life of the program,” Zanini said. “If you didn't take that approach, you'd be gambling.”

Just some food for thought for anyone who thinks agile is "unsuitable for large game projects".

Why use Scrum?

An article of mine just showed up at GameDev.Net

Cross-post: Adopting TDD

A post from my coaching blog about when it's best to adopt TDD.

Retrospective Meetings

When we talk about Agile, we use the phrase “Inspect and Adapt”. "Inspect and Adapt” is a powerful aspect of agile. We inspect the progress of the game and adapt our plans to the reality of what is emerging. This is the purpose of the Sprint Review and Planning meetings. However it also applies to how we are making the game as well. This is the purpose of the Sprint Retrospective meeting.

The Sprint Retrospective is possibly the most important and neglected practice in Scrum. The goal of the meeting is to promote continuous improvement in how the team creates value in the game. This is accomplished in the Retrospective by asking key questions about the practices that benefited the team or worked against it during the last Sprint and to identify new practices that should be started during the next Sprint.

The Retrospective is where much of the long term change of the Scrum practices occur. The changes don’t necessarily have to be large ones. A one percent improvement in the effectiveness of the team every Sprint will compound into huge improvements over the long term.

The Retrospective meeting occurs following the Sprint Review. It run by the Scrum Master and is time-boxed. The amount of time spent in the meeting is determined by the team. The meeting is for the benefit of the team but anyone can attend.

This is done by answering three questions in the meeting:

What worked well last Sprint that we should continue doing?
The practices that worked well during the previous Sprint should be identified and continued in the coming Sprint.

What didn’t work well last Sprint that we should stop doing?
The team or customers should identify practices that worked against the team during the last Sprint and focus on stopping those things during the next Sprint.

What should we start doing?
The team identifies practices that should be implemented during the coming Sprint that will help them work better together.

People outside the team can offer suggestions as well. Some in the Scrum community suggest that the meeting is only for the team, but I believe that if a team interacts with people outside the team during the Sprint, then including these people in the Retrospective is valuable.

The Scrum Master will record all the answers given for each question. Unlike the Daily Scrum, there is a lot of debate and discussion that occurs during the Retrospective. The Scrum Master should mediate these discussions to keep the meeting within it’s time-box, but otherwise allow the discussions to occur.

Action items the follow from the answers don’t necessarily have to be assigned during the Retrospective. Below are some examples of answers to the question “what should we start doing” and some potential actions that could result:

“Start having QA approve all tasks marked “done” if possible”
This doesn’t require an action item as it would be a practice for the entire team to adopt.

“Make sure Joe tests his animations before committing them”
Clearly Joe has to follow up on his animation testing, but since this has to happen daily, there doesn’t need to be an action item.

“Have the build server send an email when the build is broken”
There might be an action item for the Scrum Master to pass along this request to the team that maintains the build server. If the team has a programmer who can implement this change, then the action might be to add this request to the team’s backlog.

Following the Retrospective, the Scrum Master will post the answers given during the meeting in the team area. During the following Sprint the Scrum Master will check off all the items that have been accomplished on the list. During the following Sprint’s Retrospective, any items left unchecked from the last Retrospective and discussed. These items can be either carried forward to the next Sprint or removed from the list based on what the team decides.

Automating the Daily Scrum?

I’m often asked by teams starting Scrum “can we substitute the use of a software tool for the Daily Scrum?”. My answer is always an emphatic “No!”.

The Daily Scrum is not simply a meeting to update task hours. Its Daily Scrum’s purpose is for the team to frequently inspect their progress and make commitments to each other about the work to be done towards their agreed goal.

It takes awhile for teams new to Scrum to understand the purpose of the Daily Scrum. With their previous process, tasks may have been estimated and assigned to individuals by managers. How the tasks came together to fulfill larger goals was not the responsibility of the people working on them. Scrum turns that upside down. The team is given total responsibility for the tasks and how to accomplish the larger goal. This requires a different mindset for the team that is new to Scrum to learn. It’s not easy to learn. There are years of muscle memory to overcome. This is why the Daily Scrum seems wasteful at first. People think it’s about the task hours.

Effective Scrum teams don’t focus on the hours. Their Daily Scrum is a beehive of activity that focuses on what everyone is doing to accomplish the goal and what the problems are that need to be fixed. I call these “effective” teams rather than “experienced” teams, because I’ve seen some teams that have been using Scrum for years and never get to this level. It’s the role of the Scrum Master to push the team to being effective.

Teams that are new to Scrum that ditch the Daily Scrum in favor of a tool are potentially killing themselves at the starting gate.

Daily Scrum Question #2

Every member of a Scrum team will answer the following questions during the Daily Scrum:

1. "What have you done since the last Scrum?"
2. "What will you do between now and the next Scrum?"
3. "What got in your way of doing work?"

However, I often hear people answer the second question as if it were this one:
"What are you going to work on today?"

It doesn't seem like much of a difference, but it occurred to me that that form of the question results in answers like:
"I'm going to work more on the jump animation."
"I'm going to see if I can get that problem with the PS3 textures fixed."

There is absolutely no commitment in these answers!

I think this cascades to additional non-committal behaviors on team such as not improving task estimate accuracy or not finding ways of improving how processes work (e.g. build times, pipeline cooking times, etc).

The first two questions are meant to reflect a commitment to the team by each person on the team. It's up to the Scrum Master to get the answer to the right question.

The SM should reaffirm this with the team by reminding them in the Scrum:
"What are you committing to finishing for the team by the next time we meet? If you can't meet that commitment there should be something for you to complain about when answering the third question! It's something we all need to know, because we'll succeed or fail as a team."

Failing to meet a daily commitment should put a little bit of pressure on the person who didn't meet their commitment. They'll either get a little better about their estimates or a little bit unsatisfied with anything that slows them down. Just enough to do something about it.

Burndown Mystery # 1

What real-life story does this burndown chart tell?

NLGD - Not attending

Unfortunately due to some event management issues (it sounds like they had a major IT failure). My full travel arrangements were not passed along to me, so I ended up not traveling (I have this thing about flying to a foreign country without hotel reservations). I'll be spending time at a local developer next week instead.

I'm very disappointed. I've never been to the Netherlands, but have heard great things. Maybe next year! The clogs will have to wait...

Advice to all conference organizers: let your speakers control their itineraries.

Why...why...why?

If you have a child who is old enough to speak, you've probably been asked "Why?" thousands of times. I know I have. When my sons were ~2-3 years old, it seemed like everything I said was answered with this question. I wanted to be a good dad and not just tell them "that's just because the way it is", but explain the reasons why. They were trying to discover the detail of their new and wonderful world. It was my job to help them.

It wasn't easy. Sometimes it felt like their "whys" were a bit automated; they just wanted to hear me talk because they were learning phonemes or something. The side effect was that they forced me to dig deep into my own understanding of nature to explain something. For example, they might ask why the sky is blue. I'd tell them that the air bounces the blue colors around more than other colors. The inevitable "whys?" would keep coming. Ever try explaining Rayleigh scattering in terms a two year old would understand?

The interesting thing is that it really makes you think about your own understanding of why things are they way they are. It seems as though we ourselves reach a certain depth of understanding of the world in our own minds and then just tell ourselves "that's just because the way it is". If you've read or listened to Richard Feynman's lectures you realize that part of his genius is that he understands and can explain the way the universe works to a much higher depth of "why" than anyone else.

"Why does our pipeline suck?"

As it turns out, asking "why" is a powerful tool for creating continuous improvement in how products are made. The goal is to extend our depth of understanding about why things are the way they are and improve broken things at their root level.

Take for example, an artist sitting at their desk surfing the web during "working hours" rather than creating art. Many managers out there would be upset with the artist and tell them to get back to work. Some people using agile might say "it's up to the team to deal with how effectively each member of their team is working. Let's ignore it an see how they do at the end of the iteration."

A third choice is to ask "why":

Someone (anyone, not just a manager):
"Why aren't you making art"?

Artist (who is surfing the web):
"I'm waiting for the latest build and I can't do anything else on my PC while it's updating".

Someone:
"Why?"

Artist:
"Because the PC is very slow when the update is happening and it takes awhile".

Someone:
"Why"?

Artist:
"Because it copies the entire 4 gigs of build data every time something is changed".

Someone:
"Why"?

Artist:
"That's just the way it is".

There is a lot of value in finding the limit of our understanding of why things are and finding improvements there. Too many times we end up in situations like the one above where the artist is 50% effective because they don't know why things are the way they are. In this situation the person who does know why the entire 4 gigs of data needs to be copied over for every build might not be aware of the impact it's having.

Asking why is a tool against complacency and for continuous improvement everywhere. As Taiichi Ohno, pioneer of the Toyota Production System used to say: "The root cause of any problem is the key to a lasting solution". Ohno instructed people in Toyota to ask why up to five times....a practice that still goes on today.

What is the ideal length of an iteration?

Many things influence the amount of time that a team should have for committing to iteration goals. Some of the factors to consider are:

• Customer feedback
• Experience of the team
• The overhead of reviews and planning
• Ability to plan the iteration
• Balanced intensity

Customer Feedback

The length of the iteration depends on the amount of time the customers can go without seeing progress and providing input. Some core mechanics require a lot of feedback from customers, so maybe a two week iteration is best. Some teams don’t need such a rapid cycle of feedback (e.g. production teams) and so a four week iteration is fine for them.

Team Experience

The experience level of a team needs to be considered when selecting the length of an iteration. Teams that are new to agile often want to have the longest possible iterations. I generally discourage this because new teams will generally approach an iteration like a mini-waterfall project. They’ll spend a couple of days in design, a few weeks creating code and assets and then integrate, test and tune during the final few days of the iteration. Experienced teams will do a bit of each of these things daily which is a better way to work. It creates better results and allows the team to discover better value during the iteration. A team that is doing mini-waterfall will generally crunch at the end of the iteration and reach the finish line with their first attempt at the new or improved feature. This hardly gives them the best opportunity to create the best possible feature.

So I will often insist that teams new to agile stick to a two week iteration until they are used to it.

Overhead or reviews and planning

Imagine that you had an iteration that lasted one week. You’d probably spend one day that week in review and planning. That’s 20% of your time spent in meetings! The good news is that the overhead of planning and reviews doesn’t have 1:1 scale with the length of the iteration. A team should be able to get their review and planning done in a day regardless of the iteration lenght. Longer iterations might require a few hours of planning half-way through the iteration to evaluate the remaining tasks and their estimates.

Ability to plan the iteration

If the iteration goals have a lot of uncertainty and experimentation that needs to be done (in the form of spikes, or time-boxed tasks) then the iteration should be shorter than one with less risky goals. Risk implies that the solution might be significantly different from the one anticipated at the start of the iteration. It’s better change direction (or even backtrack) after two weeks than four in this case.

Balanced Intensity

As hard as it may seem to believe, sometimes four weeks is too long to give a team to work. This is especially true of teams that are new to agile. As described above, new teams will do mini-waterfall on iterations at first which leads to a pretty lax mini-design phase up front and an intense debug mini-crunch phase at the end. While the “mini-crunch” isn’t going to kill anyone it points to a loss of effectiveness over the entire iteration.

Teams will find what works best for them over time. For example the last team I was on chose a three week iteration because “it just felt right”. The two week iterations made it feel as though the review was too close to do anything too big and that a four week iteration was just too long. It depends on the team and the uncertainty level of their work.

When is an iteration too long?

As a customer, I’ll limit the iteration length to four weeks. Arguments can be raised that some technical goals (e.g. engine or pipeline development) cannot be completed in four weeks or show any progress in terms of value unless there is a longer iteration. My opinion is that this is an indication that the technical practices and how goals are created need to change. I’m highly concerned about any development progress that can’t demonstrate some value at least once a month.

Who selects the iteration length and when?

The iteration length needs to be decided between the customers and the team. If there is a disagreement, the Product Owner has the final word. If the team continues to believe there is a better length for the iteration, the PO should be open to trying that out. The length of the iteration can only be changed between iterations and shouldn’t change that often. Teams get used to the length of the iteration and so changing it is a bit of a disruption.

The best insider business web log

If you haven't subscribed to Keith Boesky's web log, you're missing a lot of great posts on the business model for video games. Many of the posts challenge the current business models and describe better models for the future.

I've known Keith for over a dozen years, starting with his business/legal assistance to Angel Studios. No one knows more about the game business than him. You want him on your side!

Scrum & Overtime?

Does Scrum have any rules about overtime?

Recently, the Quality of Life issue has been revived. This is a good thing to talk about regularly and I applaud Erin Hoffman, Jason Della Rocca and others for not letting this topic die. A recent Gamasutra article discusses how things have progressed since the EA Spouse days of 2004. It doesn't sound like much has improved.

I've heard people say that Scrum avoids overtime because teams can decide to drop some of their Sprint goals if they are faced with too much to do within 40 hour weeks. I disagree with this interpretation. I think it creates problems.

Teams commit to a certain amount of work that they estimate can be completed in a 2-4 week Sprint. Estimating 2-4 weeks worth of work can still have uncertainties though. There can be many unanticipated problems that can occur during the Sprint that can impact progress. But this doesn't mean the team can or should dump goals at the first sign of trouble.

Teams should take commitments seriously. This creates a pressure on them to find more effective ways to work to meet commitments. We want the teams constantly looking for small improvements. Even a 1% improvement each Sprint will add up big in the long term.

Don't Punish Overcommitment

Sometimes a team will push too aggressively on their goals or hit major problems that they couldn't anticipate or were out of their control. If we insist that they complete all their goals even if they have to work insane hours, they will not continue to push the limit on future sprints. They will sandbag to avoid the pain. Teams should know when there is too much to do in a reasonable amount of time and should go to the product owner to discuss dropping some goals when there is too much to do.

So how much overtime should a Scrum team do? It's up to them. If it's the last Sprint before a major release we'll see teams putting in a couple of weeks of late nights and rarely a weekend. If they find that they are doing this too often, they need to improve how they estimate.

One thing for sure is that when management doesn't dictate overtime to the teams, the attitude about overtime changes. When a team decides to do overtime, they do it as a team. They help each other out. It can actually be a strong team building experience.

"Sustainable Pace"

Teams should find a sustainable pace that they can work over the long term. This might mean working close to 40 hours per week most of the time with periods of crunch that don't last too long.

How long should crunch last? There have been studies that show the real effectiveness of crunch. For us, the proof came the last time management enforced company wide overtime (in 2005). We "asked" the teams to work 6 days a week at 10 hours a day to finish Darkwatch. The proof came from the velocity observed in burndown charts.

The following chart shows how much time the teams averaged in burning off their estimated hours per week:

The first week was a normal ~ 40 hour week. Weeks 2-5 were the 60 hour weeks. The first thing you notice is that the overtime in week 2 caused the velocity to greatly increase. More "stuff" was getting done! However the velocity decreased until we get to week 5 where the velocity was even less than a normal week!

How is this possible? It's simple. People get tired. They make more mistakes. They lose the incentive to create value.

This is what I love about Scrum...it creates simple empirical evidence about what works and what doesn't. This is why there is no "rule" about overtime in Scrum. There doesn't need to be a rule. If your teams are truly using Scrum to find the best way to work, they'll quickly discover that after several weeks of moderate overtime, the value is lost and you might as well not continue. It becomes common sense.

Hit or miss dead?

Before this decade, the games market was largely "hit or miss" driven. One hit would pay for a dozen failures. The reason was that games cost less than a million dollars to make and hits made tens of millions in profits. The unpredictability of success in the market worked with this model.

Over the past dozen years the cost of making games has skyrocketed. Mass market games can cost 10-30 million dollars. While sales has grown very well, it hasn't kept pace. Even the price of a game to the consumer has remained the same in real cost.

The following chart compares the average cost of developing a mass market titles (in millions) versus industry sales (taken/projected from various sources):



The result is that the ratio of hits per titles developed needed to maintain a profit has dropped. How many $10 million failures can a hit really pay for?

If this trend continues the industry for mass market games, will no longer be sustainable. How do we avoid this?

• Increase market size
  
• We've been talking about this for years. Games that appeal to a larger audience than the core gamers. Nintendo has been doing this fairly well on the Wii. Their only limit is making enough hardware!
• Increase prices of games
  
• Not going to happen. There are too many alternatives for our market to spend their money and the rental market is already an example of the push back on prices by consumers.
• Low cost games / online distribution
  
• This model is growing slowly. It's not a short term (5-10 year) solution.
  
• Reduce cost & risk of mass market games
  
• This has been a main focus of developers and publishers for awhile. Unfortunately this has led to reduced quality (shorter gameplay, more derivative titles and less innovation). Outsourcing, once seen a major area of saving, turns out to not save much cost. This has also raised discussion of improved development and production methodologies such as agile.

The industry as a whole won't fail. Not as long as there is a market and demand. The only question is what the market growth will be and what the development community will look like ten years from now.

Why Daily Scrum Meetings?

Developers who are new to scrum often question the daily team meeting known as the "daily scrum". The comments generally fall into two categories:

• It's a waste of time
• Let's use a desktop tool to replace it

I understand the comments especially from the point of view of teams that are new to scrum. New scrum teams usually have pretty boring daily scrums. They are usually reporting sessions to the scrum master about the task hours and that's it. If that's what they were all about, I'd want to automate them too!

However, I don't want teams to abandon the meeting. This is based on what the daily scrum should be. The daily scrum should be a beehive of activity where the team, who takes a level of ownership and commitment to their goals, huddles up for 15 minutes and addresses what's going on. It should be an absolutely critical meeting to have.

The worst thing that can happen is that the team abandons the daily scrum practice before they reach the state where the meeting is critical to them.

The daily scrum is about three thing:

• Goals - The team is either going to succeed or fail as a team in achieving the goals they committed to at the start of the Sprint. With games, these goals are highly subjective and subject to change. The team owns these goals and needs to revisit them daily.
  
• Commitment - Team members commit to their work in front of their peers. Often this work is based on the needs of a cross-discipline team and what it takes to let everyone work together effectively.
  
• Impediments - Cross-discipline teams should be able to fix 90% of the problems on their own. They won't get fixed if they are not make visible.
  

Here are some things the SM can do to help things along:

• Avoid the "reporting to the SM" pattern. The daily scrum isn't to report to you. Avoid eye contact (look around at everyone else or the task board). Don't write down what everyone says either. It reinforces the "task reporting" attitude.
  
• Stop side discussions, talking chickens & missing pigs. The team should agree to be there and focus for 15 minutes. You need to make sure that the meeting is as focused and effective as possible.
• Announce the start and end of the meeting. Don't let the meeting ramp up and down slowly.
  
• Dig for the impediments. If someone makes no progress on a task, there has to be an impediment. Don't let them go unreported.
• Add some relevant questions. For example, one SM I know asks their team "what is risking our sprint goals today?". This elicits questions that the team doesn't think about normally.
  

Don't be afraid to experiment

Discover what works best with your team. Reorganize your task board. Go down the list of stories rather than go around the room. Make sure you discuss this with members of the team in your retrospective before you implement these changes.

Teams new to scrum think that the daily scrum is about reporting hours. It's not. The daily scrum is a discussion on the commitments, impediments and goals of the Sprint. This requires a sense of ownership to get right, but that takes time.

Good article on adopting Scrum at "Large Animal"

An article entitled "Introducing Scrum At Large Animal Games: A Look Back at the First Year of Agile Development" was posted last week on the Gamasutra Feature Article site. It's a good read on how the studio has adopted Scrum for their projects. I especially enjoyed some of the practices they've added that create team incentives to hit their goals.

Scrum Master Qualities in Little League?

My 9 year old son's little league team just won their league's AA championship. They dominated the series and made us parents pretty proud.

As proud as I am I have to give most of the credit to the coaches. I noticed a few things they did:

• Drilled the basics of teamwork. For example, fielding a ball in practice wasn't enough. Each player had to successfully field the ball and get it to the relevant base quickly. They repeated this for every player until each of them got it right.
• They didn't favor the better players. Sure, some players couldn't pitch or didn't want to play catcher, but weaker players weren't banned to the outfield.
• The coaches were pretty quiet during the actual games. They let the players make mistakes and learn the hard way.

One thing you would notice about this team was that the boys were pretty vocal. They encouraged each other during the games. They cheered for their team mates batting. They absolutely mobbed a weaker player who made the big play with congratulations. I think it psyched the opposing team a bit to play such a vocal group that was having so much fun.

Most of all, they loved playing the game. Baseball can try the patience of a 9 year old standing in the outfield during a hot day, but these boys loved being there. My 9 year old even started collecting major league playing cards.

I think of these coaches as model Scrum Masters in a few ways. They treated the team as a team. They didn't (and couldn't) force teamwork and passion for the game...they merely employed a few practices that allowed the team to lead itself to teamwork and love for baseball Most of all, the coaches knew when to get out of the way and just let the team play the game.

I recorded (poorly) the last few minutes of the final game and posted it on YouTube. My son is # 3.
...and no, that little huddle at the end of the day isn't a daily Scrum!