Tuesday, September 18, 2012

Innovation, Velocity and Grenades

In a recent class we discussed a feature for a shooter to allow the player to throw grenades in a game as an example of how features are created on an agile project.  Questions raised were:

How do we define and document grenades up front?  

How do we estimate the cost and schedule for grenades? 

This kind of question is common, but there is another question that is less frequently asked:

"How much fun will grenades be?"

Anyone who has played Quake, Halo or CoD, knows how fun grenades can be.  Who hasn't belly-laughed at seeing an opponent run off with a plasma grenade stuck to them...both of you knowing that something bad is about to happen?

"How fun?" is a critical question and it should be balanced with the other questions.  The reason it usually isn't is because fun is uncertain.  It's a question that concerns those who are held accountable for schedules and budgets (or brushed off as a metacritic target).  As a result, schedules and budgets often trump fun.  They trump quality and as a result, lead to trumped sales.

Fun requires innovation and iteration.  Innovation and iteration can be at odds with budget and schedule.  Dramatically so.  They need to be balanced.  We've all been on projects or seen post-mortems where teams defy stakeholders and make something great  or, alternatively,  blow through years of budget and schedule goals trying to make the perfect game.  We can't leave it to a team's passion alone to decide how good a game is.  It has to be part of the business model and process.

The balance between schedule, budget and fun cannot be established in a big document up front or a Gantt chart.  It has to be iteratively guided by someone who can create a balance between development and business.  In Scrum, this the main purpose of the Product Owner (PO).

Back to Grenades

But first, let's step back and look at the life of a grenade through development:

A grenade feature might start life as a simple user story:
"As a player I want to be able to throw grenades at enemies to destroy them"

Depending on how many shooters your studio has shipped in the past, you might have more detail up front, but this is a double-edged sword because sequels often don't innovate as much.  So let's assume this is the first grenade being created.

The first sprint will focus on getting the basic physics and control in place.  We want the feature to "feel good" before we add polished assets, HUD elements or worry about how many grenades we are carrying in inventory.

During development , the team and product owner will gain knowledge about the fun, cost and schedule of grenade tossing and innovations such as "sticky plasma grenades" will come up.  Innovating and iterating on what is fun is a critical aspect of what teams do.  There is no measure for that.

Most features take longer than a single sprint to complete, but we want to minimize the amount of distance (time and cost) required to make the grenade "shippable".  This distance is called debt.   Lower debt allows balanced decisions to be made along the way based on fun, cost and schedule.  Large debt results in schedule and cost factors dominating decisions on fun.  Many of us have seen looming deadlines force us to cut features that are 90% complete.  Establishing standards of how much debt can be carried forward from sprint-to-sprint during a feature's development is useful.

Velocity, most commonly expressed as story points or ideal days of work accomplished per sprint, can create a lot of confusion and an overemphasis on velocity can collide with finding fun.

Velocity is an excellent tool for forecasting, but when it becomes the primary measure of team performance, it can become counter-productive.

Let's say we estimate that grenades at 10 story points.  Team A finishes grenades in two sprints, while team B takes three.  Is Team A a more productive team?  I can't tell you.  Maybe Team B spent extra time making grenades an innovative and fun experience.  Maybe Team B makes a game that sells 10 times as the game Team A's makes...

Velocity measurements are important, but they can't outweigh the judgement of a good Product Owner working with a motivated team.

Frequently Asked Questions
A number of questions related to this topic come up:

How much should we document and schedule up front?

When a excessively detailed grenade design and schedule is created up-front, teams are often temped to spend less time on finding the fun and rush to implement assets, HUD elements inventory details, etc because there is an assumption that the questions have been answered.  This doesn't mean we don't do any planning up front.  My rule of thumb is to start a plan on what you know about something and continually improve the plan by iterating.  Effective agile teams spend more time planning than traditional "waterfall" teams.  The difference is that agile planning is more evenly spread throughout the project.

How do we communicate cost to stakeholders if we don't know it up front?

One time I estimated it would take 2 days to tune a vehicle's tire friction model and it ended up taking 2 weeks.  It was a necessary and unanticipated cost.  The important thing is that I didn't decide to invest this time on my own.  There was a conversation between the Product Owner and me on a daily basis about how driving felt.  The Product Owner was accountable for the cost versus fun tradeoff, not me.

Same goes for the grenades:  There needs to be an ongoing conversation about the physics and control here.  Does it feel good?  How much skill is required for aiming?  Is the damage settings too high or low?  All of these questions will impact the cost and schedule.   This is where the product owner needs to weigh in.  Through prioritized course correction, all the while keeping debt as low as possible, an agile team can control quality, cost and schedule better than a non-agile team.

What's to prevent features from wandering out of control?

Agile mustn't result in a free-for-all where the team implements anything they feel like.  The Product Owner guides if-and-when innovations are taken advantage of.  For example, if our shooter takes place in World War 2, plasma grenades would not be appropriate.  Sticky bombs that use socks and grease would be.

On the other hand, maybe grenades doesn't end up feeling fun in the game and the Product Owner doesn't want to spend any more time and budget on them.

So this means we plan as little as possible up front?
No!  Let's distinguish between identifying what's possible early, which is good, from making too many decisions up front, which is not so good.   We'd rather make decision as we learn what is fun and how much it will actually cost.   When we make decisions at the start of a project in a document or schedule, we're making decisions when we have the least amount of knowledge.  That's usually when we make the wrong decisions.

Rather than try to come up with a very detailed design and task schedule that attempts to predict too much, a product owner can "cost box" and/or "time box" features.  Cost/schedule plan for grenades early in the project might be nothing more detailed than:

"We'll spend eight people-months on grenades over the next two month release.  We can't say exactly what the grenades will do right now, but we'll refine the plan as we go along and share it with you every sprint."

The stakeholders may request more detail about grenades, which might result in a few pages of detail, but over time, as teams and Product Owners build trust with the stakeholders, such demands diminish.

What other questions are you being asked (or are asking)?

Thursday, August 30, 2012

Plateauing an agile adoption

My sons plateauing in Colorado
Many studios adopting agile will "plateau" after an initial burst of improvement.  Continued improvements stall for various reasons and getting past a plateau can be a challenge.

Agile adoption is usually not too hard at first. A simple framework for iteration and daily cross-discipline conversations, all by themselves, will bring significant improvements to a studio. Unfortunately, this initial improvement is then locked-in and adoption stalls.  Agile plateaus.

What's hard about agile is building and sustaining a culture of escalating self-organization and continuous improvement.

After coaching and training teams for a decade, I've seen common plateauing patterns:

  • Lack of executive buy-in. Executives can fear that agile exonerates failure with little upside.  They hear that agile means "no planning" and that any plan, even the old, unreliable type, is better than no planning at all.  Teams are not allowed to be agile, which means that they are limited in using emergent knowledge about the game to refine a plan.
  • Middle-management accountability doesn't change. Many studios are setup so that a producer or project manager is accountable for meeting a schedule. When there is a shift to a process where teams are accountable for iteration goals and the product owner is accountable for guiding the product based on emergent value and measured velocity, accountability has to shift away from those roles. Until it does, middle-managers are going to continue to manage teams and schedules. 
  • Pushing velocity on teams. Story points must be used for forecasting (pull versus push). When they become a metric for "pushing" a team commitment, then all sorts of bad things happen. Bug-filled features, untuned assets and mechanics, low-quality code, etc. are the result. To paraphrase a certain bun-haired princess: "The more you push velocity, the more quality will slip through your fingers." 
  • Micro-managing sprint backlogs. This is similar to the accountability pattern above, but it deserves its own pattern because there are different root causes and solutions. It's simple to observe. Are teams finishing all their tasks, but delivering low value? Are there twelve different ways to analyze a sprint backlog but the team doesn't debate about making something fun? Managing the sprint backlog is very important, but the backlog must derive from quality, which is often more emergent than we'd expect. 

There are no solutions listed here for a couple of reasons: First, listing all potential solutions would require a small book because, second, solutions are different for each studios that encounters them and are usually dynamically linked when root causes are uncovered. 

Solutions are usually easy to identify but hard to implement, because they challenge a culture.  Agile is hard!

Friday, August 17, 2012

Creative Developer Motivation and Autonomy

"Motivated developers are the most productive".  Is an axiom, but as a developer, high productivity wasn't the only reason I enjoyed being motivated.  Creative motivation is intrinsic.  We are creative because we like being creative.  It gives us joy.  Making something new work is a huge reward in itself.

How do we support such motivators?  Studies have shown that the top three creative intrinsic motivators are:

  1. Autonomy - The urge to direct our own lives 
  2. Mastery - The desire to get better at something that matters 
  3. Purpose - The yearning to do what we do in service of something larger than ourselves
The first, autonomy, is a challenge in organizations because most organizations do not trust their developers to have much of it and developers don't often trust their organization back.  Trust and autonomy take time and have to be built into a culture.   Building trust often has to be built from the bottom up (unless it's part of the culture from day one).

One starting place to build trust is in the iteration/sprint.  Can developers be allowed to take control of how they achieve their iteration goals for just two weeks?   This question defines the first fork in the road between organizations that will have success being agile and those that will not.  There are many other forks in this endless, challenging road, but if trust can be built over the short term, you're off to a good start.

Monday, August 13, 2012

Axiom: Creative developer motivation and productivity

This is the first axiom in a series I'll be posting over time.

Axiom: Creative developer motivation and productivity:

The productivity of creative developers (developers who work on creative new products), is primarily based on their motivation.  Highly motivated creative developers:

  • Produce higher quality work
  • Have more creative ideas and are more likely to share them
  • Produce more (art, technology, effective designs)


axiom  (ˈæksɪəm)
 — n
1. a generally accepted proposition or principle, sanctioned by experience; maxim

I sometimes find myself in debates about the value or application of agile practices.  For example, I was speaking with a ScrumMaster who was frustrated with their team because they are not "filling in their time-sheets" or "updating the task tracking tools" they've setup to help sprint execution become "more efficient and predictable".  This led to a debate about the value and appropriate application of tools and the purpose of the team owning sprint planning and execution.  I believe such tools and practices harm Scrum teams, while the ScrumMaster believed that certainty in sprint planning and execution can be achieved with enough detail and tracking.  The root cause of the disagreement laid below the practices, in the area of core principles or axioms, for creative teams and product development.

So the idea is to start posting what I believe to be axioms about creative workers, teams and products.  The idea is to make connections to some of the practices that are debated.  This will duplicate and overlap some of the agile manifesto, which is a great statement of values, but the manifesto has been quoted so long and frequently, that we don't always think about the reasons behind it.

I'll put "axiom" in the post headings and link to this page.  I'd love to hear any comments about whether these are truly axioms, or not.

Monday, July 23, 2012

Oil tankers, publishers and blame

Years ago, my father took me sailing on Galveston Bay.  It is a beautiful place to sail, but one time I was startled to see a small sailboat like ours being hit by one of the huge oil tankers that share the bay.  The the sailboat was knocked around a bit, but fortunately it and its passengers survived.  When I asked my father about it, he told me that the tankers run over small boats "all the time".  I asked why the tankers didn't avoid them, since they move relatively slow, but he said that even at that speed, they couldn't steer fast enough to avoid them.  In fact, the tankers ended up positioning a crewman at the stern to spot and report the wreckage of any small boats they may have hit.

It seems that publishing roles can be a lot like that for large games.  A publisher producer who works with a team a few hours a week is not in a position to guide the big development effort, but is usually only able to report when the project has gone wrong.  With large document-driven projects, it's hard to see where the 80+ development team is heading or even steer them.

I've been doing more onsite coaching and training with publishers over the past few years.  At first, it was to help them work with developers using Scrum (product backlogs, user stories, roles, etc), but more recently, I've been helping them use Scrum for their own work.

Years ago, when I worked for a first-party (publisher owned) studio, I visited them to explain Scrum.  Our studio had been using it for less than a year and they were a bit concerned about what it all meant.  I spent 30 minutes describing it before a half a dozen of us went to lunch.

Lunch was an eye opener.  Everyone was complaining about the other departments and individuals in them.  Sales, marketing, QA, production, management, etc. were each, in turn, blamed for all the problems that existed.

It turns out that each of these departments were each on different floor of the building our publisher occupied and this layout didn't help communication.

I suggested that an arrangement of cross-disciplined teams each focused on a few games might help, but that suggestion was waved away as impractical.  It seemed that their daily lunchtime gripe session was cathartic to them, having given up on any hope that their company would be successful in any major way.  Blame provided an absolution of accountability.

Long-story-short, they didn't exist a year later, having been broken up and partially absorbed by a different publisher, but my thought remained; "why can't publishers organize cross-discipline teams around games?"   It makes far more sense than department organizations.  It also makes far more sense to have a cross-discipline team, meeting daily or weekly, to be responsible for a $50 million dollar game than to rely on a single part-time producer that reviews milestone builds several times a year.

A few do exist or are emerging and they are proving to be effective.  Imagine starting with a small daily/biweekly/weekly team meeting between sales, marketing, development, QA and management.  Follow this with some goals that this team would commit to accomplishing every few weeks.  Imagine having total understanding and transparency about the backlog and the velocity and understanding what the emerging game quality and development reality was telling us.  Imagine replacing blame with collaboration.

That seems like a better catharsis.

Wednesday, June 06, 2012

A Perfect Storm of Waterfall, Language Barriers and a Bad Definition of Done

Kazumasa Ebata, a talented agile coach in Japan is currently translating my book for release there.  During our conversations I mentioned that there were a few stories that we might share in the book about Japanese and American game developers working together.  Here is one of those stories.  I'm not sure he'll want to include it though.
Sometimes cultural blending works, sometimes it doesn't

In 1999, I was promoted to the role of "Director of Product Development" at Angel Studios, partly because of my role Directing Midtown Madness and partly because no one else was dumb enough to accept the job.  I took on the responsibility for seven games in development and was no longer making anything fun directly.  But I did get a better office and health care plan.

At that time, the next project Angel needed to ship was a Resident Evil 2 (RE2) port to the Nintendo 64 (N64) for Capcom in time for Christmas.  Both RE2 and the N64 were hot products and so this promised to be a great selling game....if we pulled it off.  In 1998, the video game industry was even more seasonal than today.  If you missed Christmas, you missed half your sales.

There were a few barriers to overcome.  One was a talented, yet stormy team that wasn't afraid to take on risk and challenge themselves to do the impossible.  I occasionally found myself keeping two of them apart from hitting one another.  It was our dream team.

The other barrier was working directly with Capcom in Japan.  This created a few problems.  First was the language barrier.  We understood half of what each other was saying.  The second was a culture barrier.  Capcom didn't understand why we didn't sleep under our desks when not working on the game.  The third barrier, I won't even classify.  Shinji Mikami, the legendary designer of Resident Evil, seemed to hate our guts.  He would visit us occasionally and practice scowling.  Maybe it was jet-lag or maybe he didn't like the food.  Whatever the reason, we didn't feel welcome.

Despite these barriers, we established long relationships with Capcom of Japan and Nintendo of Japan.  These relationships let us meet people in Japan that were extremely talented game developers and approached game development in an incremental, player-focused way that vastly influenced our craft as game developers.  Much of what inspired me about agile came from seeing their approach.

But back to the disaster.  The RE2 team was on track.  They were working hard to reach alpha, when all features were implemented and allowed the team to start attacking the bugs, tuning the game and polishing it.  It was a classic waterfall approach and it was headed towards the typical cliff.

We fell off that cliff in a dramatic way one sunny morning, shortly after we shipped the alpha build to Capcom.  Diego Angel (the CEO) and I received a long rambling email from a Capcom producer in Japan that I'll never forget.  The first half of the email described the general poor state of western workmanship.  The other half described how our port of RE2 fit that description.  We finally were able to decipher the problem.  Capcom's definition of alpha was different from ours.   Capcom's definition of alpha included the requirement that all bugs were found and fixed and that all optimizations were completed.  Apparently that definition was never translated and communicated to us and we never talked about it with them.

The main problem was that Capcom had no testing in place for the game.  Their alpha definition implied that we would handle all testing on our side.  Christmas was going to be missed.  We were going to deprive all those rosy-cheeked children of their zombie killing thrills on Christmas morning.

Alas, that was not the only problem we faced that morning.  As it turned out, I made it worse by committing the biggest email blunder of my life.  I intended to forward the email with a "poorly worded comment" about the Capcom producer's lack of politeness to Diego, our CEO.  Moments after I sent the email, Diego came running into my office with a shocked look: I had inadvertently cc'd the Capcom producer with the comment!

Normally this might have caused disaster, but this one time the language barrier came to my rescue.  Although the Capcom producer understood the meaning of the "poorly worded comment", he overlooked that the third-person wording of my email implied that I cc'd him by accident.  He thought I had boldly insulting him directly.  How was this better?  I don't know why, but he backed down.  From then on he was my best friend.  He brought presents from Japan every time he visited and did his best to help everyone understand that the confusion about alpha was a fault of the language barrier and not incompetence.   The relationship survived to eventually spawn the "Red Dead" series (a different strange story).

Monday, April 30, 2012

Valve's Culture, Self-Organization and Scrum

“If you don’t like change, you’re going to like irrelevance even less.” 
- General Eric Shinseki

In the spring of 2012 Valve's New Employee Handbook was leaked.  The handbook's release has led to a number of discussions about the merit of The Cabal (what Valve calls their process) and their work environment.   For me, it's hard to argue with success and everything I've read about Valve being a great place to work.

Since first reading about The Cabal in 1999 and attending a few GDC sessions since, I've been inspired.  That inspiration helped lead me to agile thinking.   I felt there was a connection with agile and the kind of place, like Valve, where I wanted to work.  A place where rigid process and hierarchies were considered a mismatch to creative development.

Valve's handbook states this belief near the start:

Hierarchy is great for maintaining predictability and repeatability. It simplifies planning and makes it easier to control a large group of people from the top down, which is why military organizations rely on it so heavily.

But when you’re an entertainment company that’s spent the last decade going out of its way to recruit the most intelligent, innovative, talented people on Earth, telling them to sit at a desk and do what they’re told obliterates 99 percent of their value. We want innovators, and that means maintaining an environment where they’ll flourish. 

The handbook goes on to describe the role of an employee in this environment.  The criticisms I've heard about The Cabal often say "you need the right kind of people for this to work".  I agree, but I think that the potential pool of such people is larger if you provide mentoring to help the transition into such an environment.  The handbook acknowledges this as a challenge:

There are a number of things we wish we were better at:
  • Helping new people find their way. We wrote this book to help, but as we said above, a book can only go so far. 
  • Mentoring people. Not just helping new people figure things out, but proactively helping people to grow in areas where they need help is something we’re organizationally not great at. Peer reviews help, but they can only go so far. 
This is a common challenge in any studio that is attempting to improve self-organization.  People have to unlearn a lifetime pattern first imposed in our public education systems, which train children how to work in a task-driven top-down hierarchical organization.  It's an even greater challenge for a studio that has been hierarchical, since it threatens the status quo (more on this later).

Self-organization and hierarchies aren't mutually exclusive.  Gabe Newell leads Valve; it's not a pure democracy, but Valve doesn't have many layers between him and an artist creating texture maps.  Nor is the artist being handed a list of texture maps he or she is assigned to create during the week.  The artist is expected to be a professional and is treated like an adult by being allowed to be personally accountable.  What self-organization does is to flatten hierarchies and reduce the number of lines of communication between people that need to communicate.

Scrum and The Cabal have the same goals
Scrum is a framework for iterative and incremental product development based around self-organizing teams.  Team size, sprint durations, Scrum roles, etc. are meant to foster self-organization.  Every 2-3 weeks, the team to inspects their work and practices and seeks ways to improve both.  The roles provide for clear interfaces and areas of ownership.  The benefit is that after awhile, every individual should find motivation in seeking these improvements.  This motivation builds on itself, accelerates and leads to a better working environment.  

Good working environments and profitability aren't exclusive.  While Valve enjoys the highest retention rates of almost any company, it also makes more profit per employee than even Google or Apple!

The goal of Scrum adoption is not to "do Scrum perfectly" but to establish a framework that will lead to such a culture.  It's been referred to as a starting script for self-organization.

Why is it so hard then?
So why do few companies ever achieve similar cultures?  Why is it so challenging for organizations that adopt the Scrum framework to become like Valve?  This is the big question.  There are many different explanations and reasons for this I've read from "people mostly suck" (a comment I saw today) to "Scrum sucks".  Both are simplistic and wrong.

I believe it mostly lies in cultural resistance.  As mentioned above, an organization that grows in a hierarchical pattern, resists the adoption of self-organization.  This applies to managers as well, who see their value tied to a command-and-control structure.  Even in the face of studio extinction, these forces resist change.  I once heard the comparison of a manager resisting change in a failing studio to that of the Titanic passenger with the finest cabin refusing to evacuate!

Resistance comes from developers as well, who focus on their tasks and discipline and leave accountability to their bosses.  This feels safe, especially in a culture that hands out blame like candy during Halloween.

Valve had the benefit of being founded with the principle above and maintaining it.  It doesn't hurt that Valve is self-funded and somewhat isolated from external customers.  It also doesn't hurt that they own their I.P.  But this doesn't mean the path to self-organization is impossible to move to from a hierarchical culture.  It's definitely hard and it takes time.

There is a revolution taking place in how we work that may take a generation to become common place.  We have more examples every year that show us how to get there and what the benefits are.

Update 1/19/14 - Valve has since posted the handbook on their website and the link above reflects that.

Tuesday, March 13, 2012

Kanban for Social & Mobile Game Teams, Part 1

Many social game teams (or MMO or mobile game teams) deploy new features once or twice a week.  For them, Scrum is more challenging to adopt because 2-5 day sprints are too short and there are often frequent interruptions to their work.  These impede the team's ability to commit to a valuable sprint goal.

For these teams, kanban is a more suitable approach.  I've used kanban for years (see links below) and have worked with many teams adopting it.  With the growing popularity of social and mobile games and their platforms, kanban has become more established.

Kanban has the following advantages:

  • It's easy to deploy.  You start by visualizing your current workflow.  No new roles are needed.
  • Deployment of features is continuous, which matches the needs of many live games.
  • It helps lead you to many continuous changes through daily inspection.
Here is an example kanban board:

If you've used a Scrum task board, you can quickly understand a kanban board.  The immediate difference you'll notice is that the kanban board has more columns.  This is because it's capturing a more predictable flow of smaller features that live games typically deploy.   For example, let's say we want to add a button which tweets a player's status in the middle of a game (by the way, I'm not a big fan of such features).  The designer specs out the button and it's behavior, it's then implemented, tested and deployed.   While this is happening, a card which represents the "Tweet Status" feature moves from left to right through the workflow.  Kanban doesn't "say" that's the best way of developing such a feature or not; we're just visualizing it. 

The other thing you'll notice is a couple of columns called "done" and the numbers at the top of most columns.  The columns labelled done are little holding buffers that we use to handle the variable amount of time for each step (e.g. some buttons take longer than others to create, so we let a little bit of backup occur so the next developer down the line isn't waiting around).  The numbers, called work-in-progress (WIP) limits, will prevent too many things from piling up.  When too many things get into the flow, there's a point at which development really bogs down (think of a Southern California freeway at rush hour).  WIP limits and a few judiciously placed buffers help things move smoothly and quickly through the flow.

The next article in this series will describe kanban's use, the blending of kanban and Scrum practices and where it makes sense.  We'll look at how to use lean practices to improve flow and handle emergencies.

Some links to previous videos, chapters or articles about kanban. (chapter 7)

Sunday, February 05, 2012

Answering the right questions

Someone recently asked how programmers and designers might communicate daily impediments if the programmers feel the designers either can't understand their impediments nor contribute to their solution.  

Designers aren't usually going to jump in and suggest a code fix, but if the programmers feel that the impediments are mainly code issues, I sense another potential problem.

The greatest challenge in making games is "finding the fun": exploring mechanics and isolating the core gameplay that will draw players.  This creates daily challenges of "how to get there".  When the primary challenge becomes how to write the best architecture or how to create the most optimized physics, then the focus is on the plan, not the game.

David Jaffe said something during the last IGDA Leader Forum that stuck with me.  He said that when developing Twisted Metal, they didn't create vehicle physics.  They created a player object that acted like a hockey puck and then pasted some vehicle models in.  Their focus was on creating the gameplay and let the player fill in the "fantasy of driving a vehicle".  

Our arcade racing games always suffered from a tug-of-war between the vehicle simulation physics and gameplay.  We should have let gameplay lead the vision.  This doesn't mean that designers have the only important work.  It means that every discipline is focused on delivering core value.  Design asks the questions and the entire team seeks the answer….daily.  Seeking those answers should create frequent, meaningful conversation.  If it doesn't, I suspect the team is answering the wrong questions.