Mixed Asset Production Pipelines & Kanban

After my GDC presentation on Kanban for production, a question came up about having a board, which models a value stream that produces a variety of assets.  It was a good question, but I didn't have the time to answer it as thoroughly as I'd like, so I thought I'd do so here.

In the presentation, I started with describing a simple flow, such as the flow of drinks at Starbucks.

Starbuck's Kanban

If you are a coffee drinker who frequents Starbucks, like me, you probably appreciate that you don't have to wait for all the lattes, cappuccinos, etc ordered ahead of you to be made first.  This is because the barista is working on those exclusively and the cashier can directly pour your coffee for you.  Looking at the Kanban board above, my coffee goes from the "order" column to the "leave" column directly.

This benefits everyone.  As I mentioned in the talk, a key metric for Starbucks is the customer cycle time: the amount of time it takes between walking in the door and when you walk out with your drink.  The critical path for coffee drinkers and latte drinkers isn't the same, but it isn't entirely separate;  Much as I personally would enjoy it, there is no separate cashier line for coffee drinkers. Starbucks has chosen not to optimize specifically for us, for good reason.

This is similar to the approach you might use for mixed asset types.  Although every asset will have a large variation of effort needed and partially separate paths, measuring every asset's cycle time will still give us valuable information.  The goal isn't to achieve a uniform cycle time for all assets; Just as people who order lattes should expect to wait longer at Starbucks than us super-efficient coffee drinkers.

Let's look at the Kanban board that shows various assets going through a production pipeline:

A mixed asset Kanban board

This board includes assets that might need particle FX or animation applied to them, or neither.  The important principles apply.  We're going to measure the throughput and limit the work-in-progress (WiP) regardless of which steps are taken.  Some assets will skip some steps like me skipping the barista.  Doing this can improve the entire system.  As a coffee drinker, I don't care how quickly the barista can make a latte, but I greatly appreciate when the under-tasked barista helps fill coffee orders.

This can happen in an asset production pipeline as well.  As we measure throughput, we can create such policies in a production pipeline:  Starbucks has far shorter coffee cycle times than barista-drink cycle times and that is fine for everyone.  The key is to measure throughput for different asset classes and explore where and when improvements for classes can improve their cycle time without impacting the other classes.

Most production pipelines are far more complex than this, but the same principles apply.  Start by simply modeling what you're doing now.  Then measure throughput and reduce WiP.

...and don't be surprised that as you try to improve your existing overloaded hetrogenous pipeline that the conclusion you arrive at is that maybe the assumptions of the pipeline need an overhaul!

Sprint commitments and forecasts

There has been much debate about whether a team, at the start of a sprint, commits to a sprint goal, or merely forecasts what they currently understand will be completed by the end of a sprint.

It's both.  In sprint planning, the team creates an initial sprint backlog, which is a forecast of the tasks, or bits of work, that they feel represents the best path to achieving the sprint goal.  The form this takes is up to them (hours, days, thrown chicken bone patterns, etc).  They will refine how they create the backlog over time to improve the value of their forecasts.

The commitment part is more about a commitment to do their best to achieve the goal while maintaining quality.

The problem is that very often this commitment comes into conflict with the initial forecast.  For example, one time I estimated it would take two days to implement drift-racing physics (with a handbrake control) into our vehicle dynamics model.  I was able to do this, but it took another week to make it "fun", much of that time sitting next to a designer.  This couldn't have been predicted and we could have stopped after two days and said "sprint goal achieved", but was it really?

At which point do we say, "it's good enough, time to move on"?  That can't come from sprint planning.  It has to come from the daily conversation with the team (including the product owner).  Sometimes this results in the forecast growing and the team delivering a part of the goal that meets the quality bar.

This definition can scare managers that first hear about it and it's where they and teams struggle at first.  This often comes from a culture that isn't prepared to trust developers to judge or achieve quality on their own and the inexperience of teams to be given this control.  So the forecast becomes the commitment and the teams focus on making the hours look good, rather than the game.   It takes time to establish the balance.

A commitment to quality at the expense of the forecast is the correct choice.  It's very easy to cut quality to look good on paper, but it will bite you in the tail end.  This doesn't mean we pursue the highest possible quality at all costs.  That quality has to be arbitrated by execution and measurement.  It has to be balanced with the needs of the customer.  It shouldn't rule at all cost.   My favorite example of "quality gone wild" comes from another driving game.  As the prototypical product owner, I encountered an artist modeling window flower boxes throughout the city players were to race in.  These required thousands of polygons and detailed textures to render.   The flower boxes were beautiful and added much color, but based on the cost of creating and rendering them, they couldn't be justified, especially from the point of view of the player, who would be passing them at over 90 MPH.

So, we killed the window boxes, but it was a good lesson on our team's path to learn how to build "95 MPH art.

Upcoming public courses in Montreal

I have two courses coming up in Montreal:

April 11-12 : Certified Scrum Product Owner for Video Game Development course.  Details here.

May 16-17 : Certified ScrumMaster for Video Game Development course.  Details here.

These courses focus on the two Scrum roles, which help guide creative teams to apply Scrum in video game development projects.  Come discuss the application of Scrum for video games with the certified trainer who has 20 years of video game development experience, who introduced the industry to Scrum in 2003 and wrote the book.

The course is open to people who develop any type of product, but benefits those who work on products that have a more creative dimension.

Agile in Embedded Software Development

In my spare time, I build various small devices using Arduino hardware or help my sons create small games.  I enjoy building devices because I had a background in hardware development as well as software development before I became a full-time game developer 20 years ago.

Embedded development benefits as much from agile practices as pure-software development.  I recently shared some tips on some of those:

• Find ways to iterate the hardware as well as the software.  We found that reducing the time between software and bring-up paid dividends despite the cost of additional hardware development.  More breadboarding/prototyping was a big benefit.
• Find ways to implement unit testing of the hardware as it's brought up and incrementally improved.  Using an example device that controls a lights brightness:  have a test that would send brightness commands to the hardware and allow someone to verify that the hardware is performing as expected at each level.  Automation of this is nice, but not always possible.
• Find ways to ensure that interfaces are established, communicated and, if changed, easily and quickly communicated between hardware and software developers.  This is usually not a big problem on small teams, but it is with larger teams.  So, for example, if hardware changes the brightness control from an analog interface to a digital one, the change is reflected in the code and tested quickly.
• Encourage hardware and software engineers to overlap as much as possible.  I like the phrase "generalizing specialists".  One tip: don't play paintball as a team building exercise.  We did that.   The hardware engineers teamed up, figured out how to increase the shot velocity of their paintball markers, and gave all of us software engineers painful welts.  It wasn't a good team building exercise. ;)
• If you have any sensors, motor controls, transmitters, receivers, etc that have to interface with the real, noisy world, try to test these as subsystems as early as possible in the target environment.  One time we went out to sea on the first test of an underwater modem which we had simulated in Matlab and an enclosed water tank.  The temperature inversion layer, multi-path and Doppler effects of the actual ocean environment demonstrated that we were very much farther away from completion than we thought.  It was a bad day.

Leading Creative, Self-Organizing Teams

On March 1st, I'll be participating in a full-day master workshop in Montreal called "Leading Creative Teams for Productivity, Innovation, and Happiness" with Christopher Avery, Jason Della Rocca and Scott Crabtree.  My part will be describing how creative teams, which have been shown to be the most productive when intrinsically motivated, can have effective leadership.

Agile frameworks emphasize "people over process"; treating people not as machines on an assembly line, but as individuals motivated by autonomy, mastery and purpose.  These motivations cannot be commanded.  They must be cultivated.

This leads to a useful metaphor to me, that a good leader of creative people is more like a gardener than a process mechanic.

A gardener cannot force a tree to grow.  They nurture it's growth.  In this metaphor, the roots are the intrinsic motivations.  Leaders cannot change these.  They cannot be forced.  They are beneath the ground: always there.

The trunk is the organization's culture.  Like a tree's trunk, this cannot be pruned or changed, overnight.  Some cultures are weak and will not support good growth.  Some are strong; enriched from the start and will provide connection between motivated people and the fruits of what they build.

The branches are process and systems.  These result from the culture, but are also directly manipulated by the leader.  Unchecked, process can grow into an unwieldy tangle of practices that sap the productivity, motivation and passion of developers.

Gardeners cannot force proper growth.  Most of what they do is subtractive (pruning, weeding, etc) and indirect (there's got to be many good fertilizer jokes here).  The actual growth results not from the gardener, but nature.  This is where the concept of self-organization and complex adaptive systems come in.  Like all other organisms,  human organizations are complex adaptive systems.  We self-organize constantly, and mostly unconsciously.  We respond to processes, system, and rules in complex, often surprising ways.  This is why there are not hit game factories and it should be no surprise that the studio that is the closest to pumping out hit games like widgets off an assembly line embraces self-organization.

So if you are in Montreal on March 1st, please join us to explore how leadership, innovation, happiness and productivity can all grow together.

Innovation, Velocity and Grenades

In a recent class we discussed a feature for a shooter to allow the player to throw grenades in a game as an example of how features are created on an agile project.  Questions raised were:

How do we define and document grenades up front?  

How do we estimate the cost and schedule for grenades? 

This kind of question is common, but there is another question that is less frequently asked:

"How much fun will grenades be?"

Anyone who has played Quake, Halo or CoD, knows how fun grenades can be.  Who hasn't belly-laughed at seeing an opponent run off with a plasma grenade stuck to them...both of you knowing that something bad is about to happen?

"How fun?" is a critical question and it should be balanced with the other questions.  The reason it usually isn't is because fun is uncertain.  It's a question that concerns those who are held accountable for schedules and budgets (or brushed off as a metacritic target).  As a result, schedules and budgets often trump fun.  They trump quality and as a result, lead to trumped sales.

Fun requires innovation and iteration.  Innovation and iteration can be at odds with budget and schedule.  Dramatically so.  They need to be balanced.  We've all been on projects or seen post-mortems where teams defy stakeholders and make something great  or, alternatively,  blow through years of budget and schedule goals trying to make the perfect game.  We can't leave it to a team's passion alone to decide how good a game is.  It has to be part of the business model and process.

The balance between schedule, budget and fun cannot be established in a big document up front or a Gantt chart.  It has to be iteratively guided by someone who can create a balance between development and business.  In Scrum, this the main purpose of the Product Owner (PO).

Back to Grenades

But first, let's step back and look at the life of a grenade through development:

Story
A grenade feature might start life as a simple user story:
"As a player I want to be able to throw grenades at enemies to destroy them"

Depending on how many shooters your studio has shipped in the past, you might have more detail up front, but this is a double-edged sword because sequels often don't innovate as much.  So let's assume this is the first grenade being created.

Iterate
The first sprint will focus on getting the basic physics and control in place.  We want the feature to "feel good" before we add polished assets, HUD elements or worry about how many grenades we are carrying in inventory.

Innovating 
During development , the team and product owner will gain knowledge about the fun, cost and schedule of grenade tossing and innovations such as "sticky plasma grenades" will come up.  Innovating and iterating on what is fun is a critical aspect of what teams do.  There is no measure for that.

Shippable
Most features take longer than a single sprint to complete, but we want to minimize the amount of distance (time and cost) required to make the grenade "shippable".  This distance is called debt.   Lower debt allows balanced decisions to be made along the way based on fun, cost and schedule.  Large debt results in schedule and cost factors dominating decisions on fun.  Many of us have seen looming deadlines force us to cut features that are 90% complete.  Establishing standards of how much debt can be carried forward from sprint-to-sprint during a feature's development is useful.

Velocity
Velocity, most commonly expressed as story points or ideal days of work accomplished per sprint, can create a lot of confusion and an overemphasis on velocity can collide with finding fun.

Velocity is an excellent tool for forecasting, but when it becomes the primary measure of team performance, it can become counter-productive.

Let's say we estimate that grenades at 10 story points.  Team A finishes grenades in two sprints, while team B takes three.  Is Team A a more productive team?  I can't tell you.  Maybe Team B spent extra time making grenades an innovative and fun experience.  Maybe Team B makes a game that sells 10 times as the game Team A's makes...

Velocity measurements are important, but they can't outweigh the judgement of a good Product Owner working with a motivated team.

Frequently Asked Questions
A number of questions related to this topic come up:

How much should we document and schedule up front?

When a excessively detailed grenade design and schedule is created up-front, teams are often temped to spend less time on finding the fun and rush to implement assets, HUD elements inventory details, etc because there is an assumption that the questions have been answered.  This doesn't mean we don't do any planning up front.  My rule of thumb is to start a plan on what you know about something and continually improve the plan by iterating.  Effective agile teams spend more time planning than traditional "waterfall" teams.  The difference is that agile planning is more evenly spread throughout the project.

How do we communicate cost to stakeholders if we don't know it up front?

One time I estimated it would take 2 days to tune a vehicle's tire friction model and it ended up taking 2 weeks.  It was a necessary and unanticipated cost.  The important thing is that I didn't decide to invest this time on my own.  There was a conversation between the Product Owner and me on a daily basis about how driving felt.  The Product Owner was accountable for the cost versus fun tradeoff, not me.

Same goes for the grenades:  There needs to be an ongoing conversation about the physics and control here.  Does it feel good?  How much skill is required for aiming?  Is the damage settings too high or low?  All of these questions will impact the cost and schedule.   This is where the product owner needs to weigh in.  Through prioritized course correction, all the while keeping debt as low as possible, an agile team can control quality, cost and schedule better than a non-agile team.

What's to prevent features from wandering out of control?

Agile mustn't result in a free-for-all where the team implements anything they feel like.  The Product Owner guides if-and-when innovations are taken advantage of.  For example, if our shooter takes place in World War 2, plasma grenades would not be appropriate.  Sticky bombs that use socks and grease would be.

On the other hand, maybe grenades doesn't end up feeling fun in the game and the Product Owner doesn't want to spend any more time and budget on them.

So this means we plan as little as possible up front?
No!  Let's distinguish between identifying what's possible early, which is good, from making too many decisions up front, which is not so good.   We'd rather make decision as we learn what is fun and how much it will actually cost.   When we make decisions at the start of a project in a document or schedule, we're making decisions when we have the least amount of knowledge.  That's usually when we make the wrong decisions.


Rather than try to come up with a very detailed design and task schedule that attempts to predict too much, a product owner can "cost box" and/or "time box" features.  Cost/schedule plan for grenades early in the project might be nothing more detailed than:

"We'll spend eight people-months on grenades over the next two month release.  We can't say exactly what the grenades will do right now, but we'll refine the plan as we go along and share it with you every sprint."

The stakeholders may request more detail about grenades, which might result in a few pages of detail, but over time, as teams and Product Owners build trust with the stakeholders, such demands diminish.


What other questions are you being asked (or are asking)?

Plateauing an agile adoption

My sons plateauing in Colorado

Many studios adopting agile will "plateau" after an initial burst of improvement.  Continued improvements stall for various reasons and getting past a plateau can be a challenge.

Agile adoption is usually not too hard at first. A simple framework for iteration and daily cross-discipline conversations, all by themselves, will bring significant improvements to a studio. Unfortunately, this initial improvement is then locked-in and adoption stalls.  Agile plateaus.

What's hard about agile is building and sustaining a culture of escalating self-organization and continuous improvement.

After coaching and training teams for a decade, I've seen common plateauing patterns:

• Lack of executive buy-in. Executives can fear that agile exonerates failure with little upside.  They hear that agile means "no planning" and that any plan, even the old, unreliable type, is better than no planning at all.  Teams are not allowed to be agile, which means that they are limited in using emergent knowledge about the game to refine a plan.
• Middle-management accountability doesn't change. Many studios are setup so that a producer or project manager is accountable for meeting a schedule. When there is a shift to a process where teams are accountable for iteration goals and the product owner is accountable for guiding the product based on emergent value and measured velocity, accountability has to shift away from those roles. Until it does, middle-managers are going to continue to manage teams and schedules. 
• Pushing velocity on teams. Story points must be used for forecasting (pull versus push). When they become a metric for "pushing" a team commitment, then all sorts of bad things happen. Bug-filled features, untuned assets and mechanics, low-quality code, etc. are the result. To paraphrase a certain bun-haired princess: "The more you push velocity, the more quality will slip through your fingers." 
• Micro-managing sprint backlogs. This is similar to the accountability pattern above, but it deserves its own pattern because there are different root causes and solutions. It's simple to observe. Are teams finishing all their tasks, but delivering low value? Are there twelve different ways to analyze a sprint backlog but the team doesn't debate about making something fun? Managing the sprint backlog is very important, but the backlog must derive from quality, which is often more emergent than we'd expect. 

There are no solutions listed here for a couple of reasons: First, listing all potential solutions would require a small book because, second, solutions are different for each studios that encounters them and are usually dynamically linked when root causes are uncovered. 

Solutions are usually easy to identify but hard to implement, because they challenge a culture.  Agile is hard!