Thursday, December 14, 2017

The Challenges of Using Story Points for Games

Story Points - The Challenges
Many game teams have used story points to measure items in the their product backlog.  There has been a lot of confusion, misuse and overuse of them.

Let’s look at some of these challenges

Using points for stories that multiple disciplines work on
Story points are meant to drive cross-functional behavior and to focus the cross-functional team on delivering increasing levels of value (velocity).  This is done by encouraging discussions about improving collaboration.

For example, if QA is a bottleneck, then the discussion turns to questions such as:
  • “How can we create functionality that doesn’t require so much QA effort?”
  • “What can the rest of the team do to alleviate this bottleneck?”
  • Etc.
Cross-functional story points are meant for complex work that is a bit unpredictable, such as developing a new mechanic or character behavior.  Teams swarm on complex work, iterating and fixing things daily.  It’s should be a true cross-discipline effort

If the work is more predictable, such as content production, then we can use functional-oriented points, or directly using time estimates with Kanban tools.  With such work, there is a flow of handoffs and we can level that flow by measuring each step (however we’re still trying to optimize the delivery of value for the entire flow, not individual steps).

Thinking story points are precise
Story points are meant to be a fast and efficient way to continually estimate and reestimate the cost of features (stories).  This helps us prioritize them and to foster conversations that don’t take up enormous amounts of time.

There is a prevalent myth that the bigger our plan, the more accurate it is.  In reality, it’s the opposite.  The better plan is the one that is frequently revisited as we learn more.  A hundred one-hour planning sessions held over two years produces better results than 100 hours of planning done up front.  Story points allow such frequent planning by being brief and focused.  An hour or two of backlog refinement every sprint is usually enough.

Translating story points to time estimates
Story points are often directly converted to time estimates.  There are a few problems with doing this:
  • Time estimates depend on who is doing which part of the story.  Any two developers can have a magnitude of difference in the amount of time it takes to complete something based on skill, experience and other factors.  Cross-functional stories don’t provide this information.
  • As rough, relative measurements, story points are not precise enough to convert to time.  This is a reason that we plan using time for a limited horizon, such as a sprint’s duration.  A story that takes 5 minutes to assign points to might take an hour to task out in sprint planning.
So what good are story points?
Story points exchange the illusion of detailed up-front planning (big documents, detailed schedules) with more frequent and brief planning sessions based on the emergent knowledge we gain over the course of developing a game.  We learn the true cost of making something fun based on execution, not documentation.

By continually updating a forecast based not just on promises made at the start of the game, but based on what reality is telling us during development allows us to better manage the outcomes.  Agile planning is better at hitting fixed dates with quality than any other approach.

The big challenge is adopting the mindset to use them correctly.  Too often our stakeholders are not aware of the power of such tools and want specific answers up front.  That’s the biggest challenge.


Unknown said...

We used to run into problems using complexity points with a cross functional team all the time. If code thinks a story is 3 points but UI or Design thinks it’s a 5 or an 8, which one do you go with? It’s the same in my new role. QA might think a story is 5 points but dev/code think it’s a 2 or 3. Do you just go with the highest?

Clinton Keith said...

Hi Paul,

This gets raised a lot.

First, I come from the school that says points are based on effort, not complexity:

But the core of your question is about cross-functional estimates.

First, I'd ask what the team is building. If they are a content production team using Kanban, then individual estimates would work fine in that flow. In this case, I wouldn't even use story points.

However, if this is a team creating new features, then thinking of points as a set of individual functional estimates isn't the best way to approach it. It represents a disaggregation of complex work into an upfront plan, when it's really a team exploration.

The goal of story points isn't to create super accurate estimates, but to drive cross-functional conversations about complex problems that they solve as a team and to address improved ways to work together.

If I were the coach (or Scrum Master) facilitating planning poker, and these numbers came up, I'd ask the group to discuss why design thinks it will take three times the amount of team effort that it would take than the programmers, not why their part takes so long. Asking it this way might reveal that the document they need to write is big. The programmers might then come back and suggest that more up-front prototyping would reduce the need to write a big document (which they reveal they don't really read anyways).

Design might just stick to their 8 points because they think there is going to be a lot of iteration and tuning for something that is easy to code up. In that case, I'd ask the rest of the team if they are OK going with the '8' and moving on. If there are a number of stories like this, then the team will start to see iteration and tuning as a bottleneck for the entire team, which might lead them to finding better workflows, tools or balancing the makeup of their teams.

iant said...

Hey Clinton,
Long time listener, first time caller ;)
I was thinking of using points to measure velocity.
Then if there is a story that I have to give to another team, I could just hand it to them without re-estimating and be confident of when they would deliver. This should work regardless of their relative velocities. If I had historical velocity data.

How do different game disciplines tend to convert effort into points?

Thanks in advance,

Clinton Keith said...

Hey Ian,

Thanks for following the blog and "calling"! ;)

The potential danger of what you suggest is that handing off an estimated story to another team filters out all the conversation that the estimators originally had. That's the primary value of that estimate. Whatever accuracy you might have with the points would be made far more uncertain by the lack of agreement with the new team.

Story points are used with stories, which are usually cross-discipline features. So the discussion will involve implementation options that have dependencies in code, art, animation, testing, design anyways. But if you get the occasional story that, for example, is a tech-focused story to refactor a part of the physical engine, I wouldn't expect the artists and animators to be involved in the discussion and estimation effort.

Hope that helps,