Thursday, April 21, 2016

Death March Crunches: 10 Causes and Solutions

There are all sorts of reasons for extended, enforced crunch (or death marches) in game development (I'll just call it crunch here).    They are all avoidable.

Below is a list of 10 of them from my experience and a brief description of proven solutions.

1. Fixed scope, schedule & cost
It’s OK to have a fixed schedule for games, but when you fix the scope as well, you’re asking for problems.  Forget about being able to plan enough to eliminate that uncertainty by remembering Hofstadter's Law: "It always takes longer than you expect, even when you take into account Hofstadter's Law."

Solution: Scope is the best and most flexible tool.  Use a prioritized scope, develop from that and derive your schedule.

Development Debt
We often build up a "debt" of work during development that is unpredictable, grows like financial debt and has to be paid back at the end through extended crunch.  There are different flavors of debt:

2. Technical debt in the form of bugs and late optimization.

Solution: address debt as soon as possible through an iterative approach that frequently delivers potentially shippable builds.  There are a number such approaches; test-driven development is a popular one.

3. Content debt in the form of a minimum required set of polished assets that have to be produced after the mechanics are created,  to tell a story or provide a minimum gameplay experience.

Solution: Build increasing fidelity prototypes and tests of assets during preproduction to measure and extrapolate the cost and schedule for full production.

4. Design debt in the form of waiting until all the technology and assets are in place before discovering whether the experience is fun enough for a full good game.

Solution:  Use iterative development is used to explore gameplay and to “find the fun” early.

5. Feature creep
We all know that the scope we ship is never what we initially planned and that feature creep is inevitable, but if there is a lot of debt being carried, it's hard to measure true progress.  This makes it hard to gauge the impact of additional features requested.  We end up backloading the game with even more "90% complete" features and therefore even more debt.

Solution: Address debt first, so that the true cost of features can be measured.  Then prioritize the feature list so that you can have conversations about where additional features can be added and what features are demoted as a result.  As long as it's understood that the feature list is not a promised set of features, but a prioritized one, where the lowest priority features can be dropped, feature creep can be managed.

6. Lack of courage
Middle management has made promises to stakeholders that can’t be kept and enforces extended crunch on development in an attempt to hide it.

Solution: Upper management needs to create a culture of transparency and not punishing those who deliver the bad news.

7. Risk adversity
Avoiding the areas of greatest uncertainty, such as unproven mechanics or new hardware in the hopes that it won't manifest.  When the risk triggers, it’s too late to do anything smart about it.

Solution: Embrace risk and prioritize work to mitigate it as early as possible.

8. Team ability
The inability of the team to make the game envisioned, due to missing skill-sets or a lack of experience.

Solution: For the short term, a risk-prioritized approach, will inform you early that the team existing cannot make the game hoped for (see courage).  For the long term grow your people through mentoring and training.

9. Ignorance of the limited benefit of crunch
For a many decades, businesses have known that crunch isn't effective past a few weeks.  The data is clear.  Sure, some extra time now and again isn't bad.  At times I've had to drag myself away from the keyboard after marathon sessions, but that doesn't mean we can or should do it for months.

Solution: As a manager who was guilty of this, I had to measure what was making it into the game, rather than just measuring how many hours people were at work.  That demonstrated the loss of value we faced with extended crunch.  That was the last time I enforced it.

10. Lack of empathy/respect
The attitude that game developers are fungible, that crunch is part of the territory if you want to make games and that they are to blame for a bad process.

Solution: Quit and work someplace that respects you.

What is missing?  What are solutions have you seen?  Let me know and I'll add it to the list (with credit to you).

Saturday, April 16, 2016

Death Marches! The evil that won't let them die.

Death marches are still an issue in the game industry.   Game developers still face death marches (excessive crunch), which leads to burnout and exodus from the industry.  I came close to leaving the industry in 2001, after the months of crunch leading up to our PS2 launch titles Midnight Club and Smugglers run.

I was shocked to see this article written by an industry veteran who suffered from "extreme exhaustion" in his earlier days and is now promoting crunch with such gems as:

"Don’t be in the game industry if you can’t love all 80 hours/week of it — you’re taking a job from somebody who would really value it."

"If working on a game for 80 hours a week for months at a time seems “strenuous” to you … practice more until you’re better at it."

Even worse, he's shared this philosophy in this horrible presentation (don't be surprised if this link is dead...I'll find another).  Who did he present this to?  How common is this attitude?

I'd write more in response, but Rami Ismail said it best.

"Sustainable Pace"

Teams should find a sustainable pace that they can work over the long term. This means working close to 40 hours per week most of the time with periods of crunch that don't last too long.

How long should crunch last? There have been studies that show the real effectiveness of crunch. For us, the proof came the last time management enforced company wide overtime in 2005. We "asked" the teams to work 6 days a week at 10 hours a day to finish Darkwatch. The proof came from the velocity observed in burndown charts.

The following chart shows how much time the teams averaged in burning off their estimated hours per week*:

The first week was a normal ~ 40 hour week. Weeks 2-5 were the 60 hour weeks. The first thing you notice is that the overtime in week 2 caused the velocity to greatly increase. More "stuff" was getting done! However the velocity decreased until we get to week 5 where the velocity was even less than a normal week!

How is this possible? It's simple. People get tired. They make more mistakes. They lose the incentive to create value.

Now, as a programmer, I love being in the "zone".  I've worked 80-hour weeks on end on a game I was very passionate about and stayed in that highly productive zone.  It never lasted more than several weeks and when I was done, I rested.  The mistake that managers made is that the effect was the cause.  They thought that forcing me to work 80-hours a week would put me in the zone.   That's like forcing someone to smile to make them happy.  It didn't work.

This is what I love about creates simple empirical evidence about what works and what doesn't. This is why there is no "rule" about overtime. There doesn't need to be a rule. If your teams are truly using iterations to find the best way to work, they'll quickly discover that after several weeks of overtime, the value is lost and that pace should not continue. It becomes common sense.

For more extensive information on this topic, check out Dan Cook's Presentation.

*The terms "velocity" and "estimation" have bad reputations in agile circles these days, but a lot of it comes from how they are used.

Tuesday, April 12, 2016

Mechanic Driven Development

I was sitting around this morning with my coffee thinking, "what the world needs is another 'X Driven Development' approach to creating products".

Well, not really, but I have been thinking about this approach for awhile and observing it in action.  This post describes a variation on how some agile game teams have changed their approach to planning and executing sprints.

A video game usually has several core mechanics that we want to develop in depth.  For example, a platformer might have movement/control, environmental challenges/enemies and power-ups/rewards.

Given this, we can start by building three branches of the product backlog:

These are core mechanics.  They are core because we have to ship with all three and they are core to the player experience.  Outside the game industry, this might be called the minimum viable feature set.  This is different.

Mechanic Driven Development (MDD) focuses on delivering a narrow set of features each sprint, which are dynamically linked to move a mechanic's playability forward.

For example, let's say we have three team each centered around a core mechanic.  The movement/control team might break down their core mechanic even further:

Now we have running, jumping and offensive moves, each broken down further.  In what order does the team develop these items?

In mechanic driven development, the question for the team becomes, "what can we do over the next iteration to move this mechanic forward?".  In other words, "what's the most important work we can do for the movement/control to make the game more playable"?

To me, running and jumping would seem to be the most important.  Other people might feel differently.  We'll talk about it as a team (including the product owner).  This conversation builds vision, creates consensus and drives developer engagement.  The resultant sprint goal contains progress of some or all of the sub-mechanics derived from that discussion.

When a team does this, the result is a sprint that is measured by the progress of the mechanics rather than a collection of stories completed, etc.

What benefits does MDD bring over more traditional approaches?
  • It engages the team in discussing the game's mechanics as a target every sprint, not just the completion of a set of stories.
  • It creates opportunities for more creative input from the team.  A ton of small user stories from the backlog often suppresses creativity.
  • It keeps the product backlog very simple
  • It makes the sprint goal very meaningful (again, instead of just a collection of stories)
  • It drastically limits design debt

MDD is a direct application of the "working software over comprehensive documentation" value in agile.  It'll often result in a direct path to playability and an elimination of excessive work called out in a detailed plan written in a design document.  This approach can also be used with a detailed design document.  We just prioritize the work and sprint deliverables that grow the mechanic and reflect the emerging game against the plan.

Give it a try!

Tuesday, April 05, 2016

User Story Mapping for Games - An Example

This article explains the basics of User Story Mapping and provides a simple example.

Years ago, I attended a lecture from Jeff Patton on User Story Mapping.  I was intrigued by his example, but struggled with how to apply it to video games.  Then, last year, he published his book on the topic.  The book is a great resource on different approaches and examples for mapping.  About this time I had a chance to mapping with a mobile game team.  Our variation on mapping worked well.

What is User Story Mapping?

User story mapping is a technique for creating a two-dimensional map of user stories usually
arranged by priority or a narrative of user activity on the horizontal axis.  The vertical axis often groups stories by sprints or releases (time).

Building the Example Map

It's useful to create a map for each release.  In a typical release, we'll usually start with a "Big Hairy Audacious Goal" or BHAG.  For our example, let's use a BHAG for a hypothetical car racing game:

"You are a fugitive driver, being chased, avoiding traffic and using your driving skills to escape Chicago!"

This is a good goal and a sizable chunk of work.  One of the first steps in release planning is to break the BHAG down into some smaller epic stories (stories still too big to fit into a sprint).

For our example, we have the following epics:
  • As a fugitive I want to drive a car fast to evade the police.
  • As a fugitive I want to avoid obstacles so I can keep ahead of the police.
  • As a fugitive I want to use the busy streets of  Chicago to lose the police.
With a bit of prioritization and refinement based on team structures, we can recreate the BHAG as a narrative made up of the epic stories:

This isn't the only possible arrangement of the narrative, but for this example, I arrange it this way to highlight the priority of the epics.  Obviously, driving the car is the most important.  It wouldn't be much fun if you can't evade the police.  Avoiding obstacles would be nice, but less important than the city and police.

Also note that the narrative isn't a set of user stories following the user story template.  The reason is that we want the narrative to be clearly readable as a whole tightly-coupled story.  The template can get it the way of doing that.

The next step is to forecast some sprints by:
  1. Splitting the narrative "epics"
  2. Sizing them by whatever method you use
  3. Prioritizing the split stories
  4. Placing them into their appropriate sprint (row) under the narrative epics (column).
In the example, the map might then look like this:

The colors represent the sprints (rows) for forecasted sprint goals.  The purple post-it in the lower right is large because it's still an epic.  We'll split that up after the spike (experimental work) on the ambient traffic in the previous sprint gives us a bit more knowledge about the work involved.

Why User Story Maps?

There are some advantages of user story mapping:
  • It visually organizes the work for a release.  I love the big picture view.  This can be used on games with a hundred people to give everyone a clear view of the shared goal and progress.
  • The narrative communicates "why" we are working on the stories.  I've noticed that teams using a map will respond to emergence better because they understand the big picture.
  • It's two-dimensional.  The one-dimensional view of a traditional product backlog is limiting.  Having multiple dimensions is better at handling prioritization.
  • It's very customizable.  For example, if you want to map a dependency, you can tack a piece of string or ribbon between two stories.
  • It responds to change very well.  We can shift stories, priorities and how work might be shared very quickly.
Other Useful Bits I've Learned
  • Build your maps with as many people as you can.  If you have 100 people on your team, have teams map out their columns individually.  This helps build a shared vision.
  • Physical boards trump tools!  I feel I'm a broken record on this, but maps don't take much space,  radiate information and can be instantly customizable and modified (sorry distributed teams).
  • Don't use post-its.  Although I use them in the art above, these maps live for months.  Index cards and cork boards, etc. work better.
Have fun mapping!

Sunday, April 03, 2016

Why Agile Game Development?

 Book Link

I made my first computer game in 1976 and became a professional game developer in 1994.  Within five years I was nearly burned out:  I had been promoted to lead seven game projects and had turned into that whip waving manager that we all hated.

But I have been inspired along the way by witnessing how people like Shigeru Miyamoto made games and what Mark Cerney wrote about his ideal process. I have also been inspired by being on a few teams that made great games and loved making them together.

This all came together when I read the first book about Scrum in 2003.  It wasn't hard to make a connection between Miyamoto's "find the fun" philosophy and Mark's preproduction experimentation approach and the values of Scrum.

So we started experimenting with Scrum in game development.  It wasn't a perfect fit.  For example, we had to go beyond Scrum for content production and support.  Along the way, we attended courses by Ken Schwaber and Mike Cohn (who also coached us onsite).  They both inspired us about the human aspect of agile.

But after using it awhile, we began to see the benefit.  Teams became more accountable.  We leaders focused less on solving daily problems for them or baby-sitting a prescriptive process.  We learned to serve their need for vision, clarity and support. Engagement, passion and fun grew.

A few years later, we were acquired by Vivendi and I started visiting their other studios to talk about how Scrum works for game development.  I also started presenting the topic at GDC to large audiences.  I enjoyed doing this and was encouraged by Mike, now a friend and mentor, to do it full-time.

So I took the leap in 2008 and began life as a one-person training crew.  I had plenty of time and barely enough savings in the first few years to finish the book.  Following that, the business became sustainable and I have loved every minute (OK, some of the airline travel hasn't been great).  I do miss working on games directly with small teams, but walking inside over 100 studios over the past eight years and getting to know the people within is rewarding.

I'm not doing this to grow a big consulting firm.  I still consider myself a game developer first and a trainer/consultant second.  However, I am a Certified Scrum Trainer and have worked with some of the most skilled agile and lean trainers and thinkers.  Combined with my game development experience this has helped me translate the purpose and values of agile and lean to the realities and challenges game developers face.

My goal isn't to ensure teams are following some rules by-the-book, but to help them find ways to make great games through iterative and human-focused approaches that work for game teams...and have a blast doing it.

Friday, April 01, 2016

The Mech Warrior 2 Hero Story

Mech Warrior 2 (MW2), released in 1995, was a big hit game for Activision and probably my all-time favorite.  What also stands out for MW2 is that the team defied a cancellation and worked nonstop to save the game while their boss was on travel.

What makes MW2 a unique memory for me, is that I finished it hours before my our first child was born five weeks early.   When my wife had early contractions, the doctor told her that if she had ten repeats within the next hour, we should to dash off to the hospital. She told me this, while I was playing the last level of WM2.  So I set the goal of completing the game within an hour.  By the time she counted to ten contractions, I had finished .  My son was born a bit premature, but healthy, a few hours later.  To her credit, my wife does not remind me of this obsessed and selfish behavior.  I blame the game.

Recently I asked a few of the participants and leaders of the original game to dig into their memory share their experiences with me.  Tim Morten, a programmer on MW2, who is now a Lead Producer at Blizzard Entertainment, shared some of that history:

“MW2 went through two rebirths: one on the engineering side, and one on the design side.  The original team had implemented something with promise, but it barely ran (not enough memory to hold more than two mechs) and it lacked narrative (just mechs on a flat surface shooting lamely at each other).  

After a couple of years of effort, with a major deadline looming, management had no option but to retrench and down-scope the project.  The existing team leadership departed at that point (lead engineers, lead producer, etc).  

In an effort to salvage the massive effort invested, a couple of remaining engineers went rogue while VP Howard Marks was away at a tradeshow for a week - without permission, they attempted to convert the game to protected mode.  This would theoretically provide access to enough memory to render a full set of mechs, but it had been deemed impossible in anything less than nine months - way more time than was available.

As of 9pm the night before Howard returned, they were ready to concede defeat: protected mode conversion requires extensive Intel assembly language programming, something they had no experience with - and there was no internet to use as a reference, they just had a single Intel tech manual.  They thought they had done the right things, but there was no telling how many bugs remained before the game loop would run.  Howard's arrival would spell the end of their effort, since his priority was to ship something, even if massive compromise in scope was required.

Against all odds, that midnight the game successfully looped in protected mode for the first time, and they were rewarded with a full set of mechs rendering - albeit in wireframe and without sound.  They were elated to have cracked the hardest problem, opening up the possibility to build a better game.

Howard returned, recognized the potential that had been unlocked, and helped set the team up for success by bringing in proven problem solvers from Pitfall: The Mayan Adventure.  John Spinale and Sean Vesce stepped in, to build a new team on the skeleton that remained, and to establish a vision for a product that to that point was nothing more than a bare bones tech demo.

The design rebirth of MW2 is something that Sean can speak better to, but it's fair to say that the technology rebirth was just an enabler - the design team innovated on so many levels under tight time pressure to produce something that was revolutionary for the time.  Without that innovation, I have no doubt that MW2 would languish in obscurity today.  Likewise, without the successful leadership of John rebuilding the team, and protecting the team from outside interference, we would not have achieved the success that we ultimately did.”

I’ve hear similar stories from numerous hit games: teams investing a measure of passion, heroic leadership protecting the team and visionary executives bucking convention and gambling on a vision.  These seem like valuable attributes to grow.  This is what “people over process” is about .