Tuesday, February 18, 2020

Ending Video Game Death Marches - #1 Managing Debt

We’ve all experienced it: A sink stacked high with dirty dishes. You rarely have time or incentive to tackle them. Usually, you’re in a rush to do it before a visitor arrives or when you run out of clean dishes for your next meal.

Eventually, most of us learn that washing the dishes once a day (or at least throwing them in the dishwasher) is a better approach. It takes a little discipline to get into the habit…similar to flossing your teeth every day, but it’s for the better good.

The same principle applies to game development. We often let the crud in our games pile up. We call this crud debt. We often push off paying that debt until it’s an emergency, and that usually leads to crunch, and lots of crunch sucks.

Why Debt contributes to Crunch (getting stuck between a rock and a hard place)
Debt is unfinished work, whether it’s bugs that need fixing, stand-in art, or untuned mechanics. It’s called debt because, like financial debt, it has an interest rate whose payback grows over time. Debt piles up —usually inside some tracking software—where it stays, growing until some point in development, commonly called alpha (the “rock”), when teams dive in to address it. The problem is that alpha is close to a ship/deployment date that is fixed (the “hard place”). Teams quickly discover that there is not enough time to address all that debt, and management decides it’s time to crunch.
It Often Gets Worse 
Ironically, managers usually react to crunch, low quality, and missed deadlines by demanding more detailed task planning. This tends to squeeze out the slack that should be used to address emergent debt. That slack is essential; when was the last time you estimated the amount of work it took to fix a bug that hadn’t been found?

Setting Aside Time to Manage Debt Works, but it’s Not as Easy as You Might Think
It’s easy to see that if you have less debt, there’s less reason to crunch because of it. So what’s so hard about managing debt?
There are two significant reasons that we fail at managing debt. First, it seemly slows down development. Practices like unit testing, creating automated test tools, and making sure new features introduce minimal debt takes extra time upfront, but in the long run, it saves a lot of time. However, it’s often hard to convince a stakeholder that doing the extra work now is a lot less expensive than doing it months from now.
Second, we developers can be lazy. Yes, I said it. Managing debt, such as optimizing polygon usage, replacing stand-in audio, or refactoring code that has gotten “crufty” is a lot less fun than tossing in something new. Like washing your dishes daily, it’s less fun than making them dirty, but it’s a discipline that needs to be built up to avoid the piles in the sink.

Where to Start
Debt is best managed by setting aside time to eliminate levels of it. You can start by establishing an agreed-upon time that’s set aside to address debt. For example:
  • Every day, eliminate any problems that cause the game to crash or otherwise be unplayable.
  • Every Sprint, make sure the game is demo-able to internal stakeholders. This could mean that the frame rate is stable and fast, and players can have fun.
  • Every Release or milestone, make the game worthy of showing the outside world. It could be missing key features, like a marketing demo, but what is there is polished.
How to Master It
  • Build test automation. The Gear Up book describes several useful ways automation can help
  • Metrics. There’s a lot of debate about whether tools that measure code quality, like unit test coverage, help. I feel they do. If you can measure something useful, you can improve it, but beware of the metric becoming the goal.
  • Educate. Try pair programming. For example, hold a “Wednesday Pizza Talk” (Gear Up) to educate developers about practices to reduce debt.
Learn More

Monday, February 10, 2020

Six Signs your Game is in Trouble

We all want to make great games and not suffer making them, but sometimes it doesn't work out that way. Below is a list of 6 typical signs that the game you're working on is in trouble.

1. Your bug database is growing out of control
I'm not a fan of bug databases to begin with. They are often rugs to sweep dirt under, and that dirt gets more expensive to clean over time. All that debt has to be paid off, and it's often paid off with crunch and compromise. We should be spending time at the end fine-tuning the experience of the game, not making it barely shippable.

2. You Don't See the Big Picture
Often, especially on large games, individual developers don't understand the game they are working on. It's all in the head of some lead designer, who might even be in a different city. Without a shared vision, moving your game forward becomes like pulling a wagon with a hundred harnessed cats. Chaos ensues.

3. Gantt Charts
Detailed Gantt charts often just serve as project management theater. A complex, graphical chart can often placate a publisher. Not that it's terrible to think about the complexities of work and dependencies, but these artifacts, adopted from manufacturing, aren't well suited for creative work. For one, they don't lend themselves to change very well. The one thing I always look out for are Gantt charts that magically slope downward off in the future when some magical burst of productivity is forecasted, which brings us to the next sign.

4. Wishful Thinking
I'm an optimist, but it usually takes the form of "yes, I'm positive something will go wrong!" Projects should embrace risk. It's more important than task management. Problems do not solve themselves and that reassuring management reserve set aside for problems will probably be gone by the time it's needed. If the game is not fun and on track now, it's unlikely it will suddenly become so "someday."

5. Building Final Content Based on Unproven Technology
This is probably the most expensive one. Technical promises and their schedules are often not worth the electrons used to store them in the project management tool. Even console manufacturers are guilty of this (who remembers the "Emotion Engine"?). If you are creating final, shippable content using budgets (graphics, physics, AI, etc.) that are beyond what your current engine can do, it's a good sign you'll be redoing all that work, in crunch, again.

6. Management "Tells"
As with poker, there are some tells that managers show, which are signs that there is trouble.

  • Mandated crunch. Scheduled crunch often means more is coming. It's panic time.
  • Moving people between projects to speed things up. This always slows things down. You know why.
  • Sudden micromanagement. I'm all for management being engaged daily with developers and communicating more, but when it instantly ramps up, they're worried.

This might sound a bit negative, but game teams get into trouble all the time, and this list could be much more extensive. Identifying the signs early on is the first step in solving them.

Well-proven solutions to these problems exist, but they are not easy. They involve changing approaches to how we think of game development and stakeholders. We need to focus on the game first and the project second. That requires courage from leadership and developers.

Friday, January 24, 2020

Solving Large Team Dependencies

Simulated Annealing for a Travelling Salesman

We've all seen it. The larger a game team, the more dependencies between developers and teams emerge to slow development down to a crawl.

The problem of dependencies is a complex one. They are called NP-complete" problems, usually only solved by time-consuming brute force approaches. So forget about an easily managed solution.

The best approach has its analogy in computer science called "simulated annealing," a technique where you start with an approximate solution and add a bit of change from time to time and see if it improves the solution. The GiF above shows simulated annealing as applied to the classic Travelling Salesman Problem. Instead of thinking of cities (groups of dots), and paths between (lines), think of developers(dots), teams(groups) and dependencies (lines). Over time, as teams and individuals within teams experiment with ways to reduce dependencies, you see those inter-team dependencies reduce.

For large teams, those changes are often to team makeup and the formation of the Product Backlog. The goal isn't to eliminate inter-team dependencies, but to move as many as you can within the individual cross-functional teams (Scrum-sized, 5-9 developers). Within those teams, they build accountability and better practices to address debt and reduce the cost of dependencies.

By experimenting with more self-contained teams and organizing the Product Backlog to reflect those teams and to depend on fewer "future integrations" and build development practices, dependencies will slowly diminish over time.

Practices to Try

  • Instead of creating a detailed release plan every few months, just define the major epics and let teams reform around the epics they'll take on and refine the release plan themselves.
  • Identify dependent specialties in the release plan between each team instead of tasking out dependencies. Often this indicates where you don't have enough specialists or where there is an opportunity to cross-train someone and spread some skill.
  • Talk about them in retrospectives. Encourage the team to come up with solutions.
  • Measure inter-team dependencies. If you don't measure it, it's harder to improve it.
  • Find ways to visualize dependencies. Program boards that use strings to identify inter-team dependencies are useful. A board that looks like the one below should horrify anyone.

This is not a good "after" shot IMO

Dependencies on large games are a huge anchor that slows development down in a very opaque way. Your focus should be on those and less on tracking individual effort.

Tuesday, November 19, 2019

Too Many Rules

A battle among homeowners is a lesson for studio leaders.

We live in a neighborhood in Colorado that borders a vast national forest.  We have lots of dirt roads, utilities, a lake, and dense forest, which require a great deal of work to maintain.  Volunteer neighbors have done much of this work, but over the past few years, volunteerism has declined.

This community also has a Home Owner Association (HOA) that likes to pass many rules.  For instance, you need to fill out a form to move a rock in the common area. If you don't wear non-flammable clothes while using an outside fire-pit, you are subject to a fine.

I see a connection between the declining volunteerism and the rising number of governing rules. This connection crystallized during a recent mountain bike ride.

I was riding on a world-famous trail network called "Phil's World" in Cortez, Colorado. Phil's contains dozens of miles of amazing trails. It is continually being expanded and maintained by a group of local volunteers that donate days of their time each month.  These volunteers will not only move rocks to clear paths, but build impressive cairns with them.

This effort inspires me to stop my bike and move rocks or pick up any litter I find. Although Phil's is an hour drive away, I feel part of this community.  I feel it more than the one where I live.

Intrinsic Motivation

Dan Pink's book "Drive" explains a lot about why we can work so hard at things we don't get paid for and so little when we are. He describes three factors of intrinsic motivation, backed up by decades of research, to prove it:

  • Autonomy — Our desire to be self-directed. It increases engagement over compliance.
  • Mastery — The urge to get better skills.
  • Purpose — The desire to do something that has meaning and is important.

When leaders or governing bodies define a broad set of rules and penalties for non-compliance, it undermines a sense of autonomy.  Such compliance is a poor substitute for purpose.

Rules and penalties often exist to force behavior that you're not trusted to do on your own. When someone tells me that they'll fine me when I wear the wrong clothes by my fire-pit, I'm less motivated to help them cut a tree down.

Excessive rule-making happens in studios, and the same lack of motivation occurs. The prevailing  attitude becomes "I did my tasks, if the game is broken its someone else's problem." Leaders subsequently get frustrated with the lack of accountability and generate more rules.

A certain number of rules are necessary, but a good practice is to engage those you are making rules for. "Working agreements" are often better. Find ways where purpose can drive behavior instead. Sprint Goals and a definition of done are useful tools for this.  Collaboratively crafted, they drive behavior better than assigning work and creating rules about how disciplines should interface and test.

Wednesday, November 06, 2019

Sprint Commitment?

I recently finished the first draft of the new edition of "Agile Game Development." One of my reviewers commented that I still use the phrase "Sprint Commitment" after its removal from the Scrum Guide, but I left it in there.

One of the reasons that the phrase is no longer in the Sprint Guide is because it has often been weaponized to force teams to complete everything they'd estimated in Sprint Planning regardless of what problems emerge during the Sprint. This has resulted in teams compromising quality to get every feature in the game "done" by the end of the Sprint.

The Sprint Guide now uses the word "forecast" instead of commitment for Sprints, even though "commitment" is still a core value. I agree that forecast is an accurate way to express what the plan is, but I think we've lost something by dumping the word commitment.

It comes down to what we mean by commitment. Often forecast means "we'll get done what we get done," or better yet, "we'll do our best to get everything done, but we might have to drop some less important features." To me, commitment goes a step beyond that, but not so far to mean "we must do everything."

Commitment means doing our best to achieve a goal, but also being accountable as a team, when things don't go according to plan.

A good example is when you commit to picking up your child at daycare. You'll obviously do your best to be there on time, but sometimes things go wrong. Let's say your car breaks down a few miles away from daycare. Do you just say, "oh well, I tried my best"? No! You call your spouse, friend, and/or the daycare to let them know. You do this because you hold yourself accountable to pick your child up.

Similarly, if a team runs into problems with a feature in their Sprint goal, they need to hold themselves accountable. They need to raise the issue. They grab the Product Owner and discuss ways to address it. If they can't solve the problem themselves, they recruit the Scrum Master to help out. It still might mean the feature gets dropped, but the accountability results in risk being managed better.

Commitment is a core value for Scrum Masters to grow with a team. Sometimes the first step is to stop solving problems for the team and start asking them, "what should you do about it?"

Sunday, June 23, 2019

The Death of Projects

Game development used to all be "project-based" . Projects have discrete beginnings and endings  aimed at developing a game which was then released, completely debugged on a cartridge or disc. Following this, a team would then move on to another project. As a developers, I always felt a bit of sadness at shipping a game I'd worked on for many months and starting fresh from scratch on a new one.

With the spread of the internet in the 90's, PC games had the advantage of allowing upgrades and expansions after a release and a few developers stayed with a game for its entire life. Some popular MMOs have support crews fixing bugs or making sure the game keeps up with the latest OS changes many years after their initial release. Digital distribution platforms such as Steam have built upon the PC's strength of connectivity and openness to allow many developers access to popular online marketplaces with incrementally released games.

As consoles gained internet connectivity and local storage, we started seeing a similar adoption. Often, it wasn't a good thing for players. Day one patches became a way for game developers to push off bug fixes and optimizations. This forced players to wait hours for updates to be downloaded before they could play a stable and polished version of the game.

Mobile games have pioneered the live game model and open marketplaces by offering cheap or even free versions of their games, which are monetized through advertising or by selling add-ons which enhance the value of the game, etc. Console and PC games have been slower to catch-up with this model, but as players are becoming accustomed to the expectation of continual growth of their games, it seems the market is changing to adapt to it.

As games and teams transition to live models, the term "project" has less meaning. Schedules, budgets and feature specifications become more fluid based on the day-to-day key performance metrics rather than fixed up-front profit and cost projections (aka guesses) from a marketing group. Rather than the approach of stuffing every conceivable feature into a single release and hoping that some of those features will drive sales, many AAA game developers are approaching a "Games as a Service" (GAAS) model of releasing a Minimally Playable Game (MPG - a variation of the Minimally Viable Product approach) and keeping teams together to release add-on features based on how the market reacts.

The project approach may be coming to an end.

Friday, June 07, 2019

Sprint planning and estimation are not the same thing

I run 10-minute sprint simulations in my classes and notice something consistent. When left to their own devices, teams will plan out the sprint but not estimate much.

For a while, that bothered me.  Weren't they listening in the class?  Then it occurred to me: when I work on my own projects, I plan out what I hope to accomplish, but I don't estimate much either.

OK, so I don't have a deadline or contract with my own work and neither do my course attendees.  Maybe that's the difference.

But then again, we're talking about sprints, which are time-boxed. We know when they end.  We also know a sprint's input, the sprint goal, and the output, which are the new features added to an improved game. We typically forecast the future progress of the game by measuring the average output per sprint and multiplying it by the number of sprints remaining.

Planning a sprint is usually a discussion of what a feature needs to do and the work needed to implement it. This is often followed by creating tasks such as "model the character"or "rig the model", etc. This is a useful discussion which reveals questions with the goal and generates discussion.

In addition, teams will put often estimates on those tasks, usually in time. There are a number of typical reasons for this:

  1. It can further discussion by revealing misunderstandings about the effort and complexity of a task and help refine the goal. Many times I've heard large numbers being thrown out which revealed a misunderstanding of the existing tools or tech or a shortage of a certain discipline's capacity.
  2. Estimates are used to create a forecast of the work the team will commit to achieving with the Product Owner.
  3. Estimates are used to track team progress throughout the sprint.
I see the value in #1, but often see #2 and #3 being misused. This reflects back to the Scrum values of commitment and respect:
  • Commitment: This doesn't mean the team commits to a fixed amount of work. Work is often unpredictable and iterative. When we aim for a fixed amount of work every time, we often have to compromise quality to get there. Commitment is more about doing our best work to achieve a goal, but also committing to doing it with quality.
  • Respect: You can't respect those you don't trust.  Too many times I see teams being tracked using their own estimates and being punished if those estimates are incorrect. All this leads to is padding, disengagement, debt and compromised quality.
Try this
If you truly want to measure progress, measure just the output of the team. Let the team decide how to plan and at what level to estimate. It can take some courage, like when I finally had to hand over the car keys to my son for the first time. But as with him, I had to make sure he was first ready. That's my job as a dad. Similarly the job of coaches and leaders is prepare their teams for higher levels of maturity and self-organization.

Sure, some teams might not work well together and their output will show it. It's far better to coach teams on creating better value than creating better estimates.