Wednesday, December 18, 2013

Managing Dependencies

The inability to manage dependencies on a large game project can derail it quickly.  Dependencies must be managed.  The question is how.  Traditionally managed projects often attempt to identify and manage dependencies at the start of a project: often creating massive Gantt charts in the process that look very impressive, but fail to anticipate the uncertainties with schedule and allocation very well.

Large agile projects manage dependencies differently.  Identifying big dependencies and risks up front is a useful exercise, but it’s not enough.  Managing dependencies on a large scale agile project follows the pattern and mindset of most agile planning: to frequently inspect and adapt and respond to change.   Agile game teams do this through well formed user stories, cross-discipline feature teams, breaking through knowledge barriers, emergent practices, release planning and emergent production planning.

Well Formed User Stories
As suggested by the first part of the excellent INVEST mnemonic, user stories should be independent.  Independence means two things; First, try to keep stories from forcing an order of development.  Stories should be self-contained measures of value that stand on their own.  Second, stories should contain no hidden dependencies that would result in similar stories being wildly different in size and cost.  For example, consider the two stories below:
“As a player, I want to smash open doors to get inside buildings”
“As a player, I want to smash open crates to find any valuables they contain”

Both stories look similar, but let’s say there is no destructible prop technology in place.  That means the team that takes one of these stories first would need to implement that technology as well.  As a result, they’ll spend far more time implementing the story than the team that takes on the second one.  This kind of hidden dependency makes planning and forecasting more challenging.  There are two ways to address it:
  1. Combine the two stories, with the technology work, into one story.  This works if the resulting story fits into one sprint.
  2. Split the destructible prop technology work out into a separate story.  Now, you’re going to say “Wait!  That creates a dependency!”  Yes, it does.  I would prefer to include one of two props (door or crate) as part of that story, but there still is an order of implementation  created that explicitly calls out the technical dependency.  Sometimes it can’t be helped.
Good user stories is a starting place for the practices below.

Cross-Discipline Feature Teams and Breaking Through Knowledge Barriers
Cross-discipline teams contain everyone necessary to implement well-formed independent user stories.  These teams address cross-discipline dependencies on a daily basis.  Working closely with other disciplines to achieve a shared sprint goal leads to occurrences of “The Medici Effect”, where breakthroughs come from related areas of work.  For example, one team had a challenging dependency between the animators and audio composers that wastefully forced the composers to wait for completed and polished animation sets.  They tried to solve this between themselves with little improvements.  A tools programmer heard them talking about it and suggested a different workflow that could be supported by a quick tool change.  This completely eliminated the dependency.  I was most impressed not by the tool change, but by the fact that an idea for a better workflow for animators and composers came from a different discipline.

The focus on adding value to the game every sprint will also elevate knowledge within each discipline.  If I’m a programmer on a team that is having problems with the character controller, and I didn’t write it, I’ll seek out other programmers who know more and learn from them about how to solve such problems.

Cross-functional teams solve about 90% of their dependencies on their own.   These are mostly
smaller dependencies.  The following practices address the larger ones.

Emergent Practices
Detailed design documentation and technical architectural specs have their place.  Unfortunately, that place isn’t for areas that are uncertain.  We can’t plan away uncertainly, we have to learn and execute it away.

Dependencies proliferate from large documented plans.  We invest our faith in them and create “resource allocation” spreadsheets that attempt to faithfully implement a plan and this creates a “chain of dependencies” that fit together exactly, like the parts of a precision Swiss watch.  Unfortunately uncertainty erodes that faith through delay, rework, waste and complexity.  This knocks resource allocation plans further and further out of alignment.  As a result, the dependency chain starts to fail.  These failures are largely the source of the “how do we manage dependencies on large projects” angst.

Agile projects handle this by not falling into the trap of false certainty.  The primary means to this is though so-called emergent practices.  These practices such as Test Driven Development and their analogs for design and art, which, boiled down to their essence, mean:

You do the least possible to achieve a short term goal and as the true value and needs for a longer term goal emerge, you build upon the work, refactoring it to meet the emergent requirements and quality bar.  Through this, you find the shortest path to the best goal.

There are simpler ways of saying this.  A couple are:
The bottom line is that uncertainty can’t be planned away.  Our practices must reflect the level of certainty we actually have.

Release Planning
Most of the dependencies that cross-discipline teams can’t handle can be addressed in release planning.  These include dependencies that exist within large stories (epics) and specialist dependencies.

The example above identified  the destructible prop technology as a dependency for destructible doors and crates.  If the technology takes a full sprint, or longer to implement or needs to be implemented by another team or specialist, it creates a dependency.

Release planning can identify these dependencies and factor them into the release plan.  A release plan is a set of speculative sprint goal forecasts covering several months, which is updated every sprint as knowledge, cost and value emerge.  As at the word “forecast” implies, it’s never considered concrete and final, but has an accuracy which reflects the actual certainty present.

Even large teams will have a few areas that are only staffed by one or two specialists.  Examples include full-screen effects artists and low-level programmers, etc.  They might not even be needed full-time, but when they are, they are often needed by one or more teams.  Even finding ways to educate other developers to off-load some work, they can be the source of bottlenecks and dependencies.

Release planning can’t plan away the fact that something might not be completed on time or that you need to hire more specialists or cross-train more.  It just creates transparency, which exposes those problems for you to solve.

Emergent Production Planning
Video game development often has a major part that can’t be as agile as other products: asset production.  Whether your game has to ship with 32 sports teams/stadiums or a minimum of 12 hours of gameplay, you need to produce a certain amount of content.  This content is dependent on gameplay mechanics, metrics, technical and financial budgets and possibly a fixed ship date.  This creates a number of dependencies and risks.

This is why developers separate out a pre-production phase, which is highly fluid and agile, and production:  a more certain and predictable phase used to develop assets .
Unfortunately, we often treat the transition from pre-production to production as a fixed date.  It should be treated as crossing a boundary from uncertainty to less uncertainty.   This boundary is defined by the knowledge we have about production dependencies and risks.  These might include:
  • What is our polygon and texture budget for static geometry?
  • What are the budgets for dynamic objects, including NPCs?
  • What fits on the disc?
  • What gameplay mechanics are assets dependent on(e.g. jump height, AI count, etc)?
  • How much does each asset cost to make and how long does it take to make them?
These questions must be answered in pre-produçtion through the completion of user stories that bring us closer to the answer every sprint.  Examples of such stories for levels would be:

“As a level designer I want a low-resolution level that represents the full extent of a production level’s size so I can experiment with the pacing and some initial vocabulary of the mechanics.”

“As a level artist, I want to prototype a quarter of a level with production-quality assets, so I can learn about the cost and quality of producing full levels.”

With a lot of money and careers at risk, stakeholders often want all questions answered at the start of a game development project and I’ve never found it acceptable to tell them “I don’t know…we’re agile”.  I’ve found that explaining (yes, sometimes in a document) what all the known risks were, how we’re going to identify when each are triggered, and a resulting mitigation plan was a good replacement.  This book provides a good reference to that approach.

Dealing with dependencies on an agile team requires the same mindset that all uncertainty does: to attack it head-on through inspection and adaptation.  Reality trumps all plans, sometimes in nasty ways, but we have to embrace it and “waltz with the bears”.

Saturday, December 14, 2013

Scaling teams isn't very safe or easy

When I started creating games professionally in 1993, our largest teams contained about a dozen
developers.  The roles of producer and designer were new and some questioned the need of having either.  Games took a year or two to develop and cost upwards of five hundred thousand dollars.
The times have certainly changed.  Hardware capacity has followed the growth identified by Gordon Moore in 1965, and software and asset complexity have had to keep pace.  Staff sizes and budgets have grown almost logarithmically, while development cycle time has been kept to within a year or two.  This has created unique challenges and pressures that collide with human nature and project management realities:

  • Larger groups of people don’t communicate and work as effectively as smaller groups.
  • Adding money and people to keep a project on schedule is one of the worst things you can do to the project and its schedule.

Managers don’t ignore these realities because they are not understood, but because the market demands more games and businesses demand more profit.  Without better tools, these demands put enough pressure on them to make bad decisions.

But there exists tools to deal with these realities and provide answers.  How can we make games with small groups of developers that communicate well?  How can we maintain a schedule with a fixed ship date and not compromise quality?

The answers lie in leveraging the strengths of people and communication.  They require us to balance the planning and inspection of an emergent game rather than relying too much on a speculative, but highly detailed plan.  They demand that we focus on the most important parts of our game first and develop them to the point where we can answer questions that must be answered before we move on to less important parts.  They insist we include stakeholders in our frequent observation of the game and plan refinement, rather than pushing them off to a post-alpha wilderness, where their voice is too distant to have much influence.

The answer does not lie in feeling “safe” by creating an assembly line of rules and processes meant to be a one-size-fits-all approach applied to everything from banking software to interactive entertainment.  It requires constant vigilance to balance quality and market needs and to fight chaos and uncertainty head on.  It’s hard work, but it’s rewarding and we love it.