Thursday, January 22, 2009

Waterfall Game Development

In the dawn of video game development, a single person working on a game didn’t need much in the way of a “development methodology”. A game could be quickly developed in mere months. As the video game hardware became more complex, the cost to create games rose. Within a decade, instead of taking three or four person months to create a game, a game might take 30 or 40 people-months. Software and art production became the greater part of the cost of a releasing a game to market.

In response to the increasing risk, many companies adopted waterfall methodologies in attempt to reduce risk. Waterfall is forever associated with a famous 1970 paper by Winston Royce. The waterfall methodology employed the idea of developing a large software project through a series of phases. Each phase led to a subsequent phase more expensive than the previous. The initial phases consisted of writing plans, which gave detail to what and how the software would be built. The middle phase required writing the software. The final phase was integrating all the software components and testing the software. The idea was that each phase reduced risk before moving on to the following phases that were more expensive. A project would never return to a previous phase once that phase was complete.

Waterfall describes a strict one-way flow of phases; once design is done, a project moves to the code phase and there will be no more design. In reality, no game developers use this method based on its original definition. Instead, most games use a sequential phase approach to game development. This approach weighs each phase sequentially in time (e.g. we do more concept work at the start and more debugging at the end), but does not impose the one-way flow that Royce’s model does.

Ironically, Royce’s famous paper illustrated how this process would lead to project failure. In fact he never even used the term “waterfall”. However the association stuck; when we use the term “waterfall” we are really referring to a sequential phase methodology. So don't bother adding a comment that "waterfall really isn't waterfall". We know!

Monday, January 19, 2009

The Golden Age of Iteration

In the golden age of the video arcade, during he late seventies and early eighties, games like Pac Man, Asteroids, Centipede and Defender were gold mines. A single $3000 arcade machine could collect more than $1000 a weekend in quarters. This new gold rush attracted quite a few prospectors. Many of these “wanna-be” arcade game creators lost their shirt in the rush to release games. A manufacturing run of 1000 arcade machines requires a considerable investment; a poor game burned into these machines would return little on that investment.

Faced with millions of dollars of potential investment, arcade game developers sought to insure that the best possible game software was developed. Games themselves cost less than a single arcade box, so it was highly cost effective to throw bad games out and try again and again before committing to manufacturing hardware. As a result, game software development was highly iterative. Executives would fund a game idea for a month of development. At the end of the month they would play the game and decide whether to fund another month or move to field testing.

Companies like Atari would field test a game idea by placing a mocked up production machine in an arcade along-side other games. Several days later they would count the quarters that the machine collected and — based on the money collected — decide whether to mass produce the game, tweak it or cancel it outright. Some early prototypes were so successful that their coin collection boxes overflowed and led to a failure of the hardware!

This iterative approach help fuel the release of consistently high quality games from companies like Atari. The decline of Atari and the market in general in the mid eighties was caused by the larger proportion of inferior games released due to dropping hardware costs (through the introduction of cartridge based consoles). The financial “kill gate” of distribution cost disappeared and so did much of the disciplined iteration that insured better games.

Monday, January 12, 2009

Traps & Pitfalls of Agile Software Development - A Non-Contrarian View

Sean Landis and the Salt Lake Agile Roundtable created a good list of traps and pitfalls of agile.

"Contrarians reject the majority opinion and often have personal reasons to degrade Agile. These writings often disqualify themselves through bias, thus failing to benefit the community. But non-contrarians - practitioners who tend to support the majority view - may be in the best position to drag the skeletons out of the closet."

1. Agile teams may be prone to rapid accumulation of technical debt. The accrual of technical debt can occur in a variety of ways. In a rush to completion, Iterative development is left out. Pieces get built (Incremental development) but rarely reworked. Design gets left out, possibly as a backlash to BDUF. In a rush to get started building software, sometimes preliminary design work is insufficient. Possibly too much hope is placed in refactoring. Refactoring gets left out. Refactoring is another form of rework that often is ignored in the rush to complete. In summary, the team may move too fast for it's own good.

This is a common problem discussed earlier here. Games can be impacted by design and art debt as well as technical debt.

2. Successful Agile development presupposes that team members will all act like adults. That's a euphemism for being competent and professional. Agile teams are expected to to accept a high level of responsibility and accountability. When they don't, things can fall apart really fast.

Don't expect everyone on the team to have the same level of competency and professionalism though. Teams need to have a mix. Teams will identify leaders that can mentor others on the team. However some teams can lack leadership and get stuck in the "storming stage".

3. Agile methodologies misunderstood may lead to team burnout due to an irrational culture of urgency. We even use the word sprint, a term that is not connotative of sustainability.

I really haven't heard of this or seen it. I suppose that we're a little battle hardened when it comes to crunch, so the constant level of urgency we see on Scrum teams doesn't seem like a big deal.

4. Agile requires more team and individual discipline, commitment and openness than a dysfunctional organization may be able to bear. Yet a dysfunctional organization is likely to place excessive hope in Agile as a silver bullet.

Oh yeah. This is a common one.

5. The high visibility on agile teams causes poor performers to stand out. The benefit is that the organization can take corrective action. In the absence of corrective action, poor performers may try forms of sabotage to avoid detection.

One of the biggest types of mistake I made would be "wishful thinking" about poor performers. These would be the people that team after team would reject. I would hold out hope for them to "turn things around"; maybe because they were smart or passionate about making games. In the end they always left (on their own or finally fired), but it always did more damage than it would have had I acted sooner.

You have to pull the band-aid off quickly to suffer the least pain.

6. Agile teams have a tendency to focus on tactical accomplishments at the expense of strategic goals. Missing the big picture can lead to long-term failure at the expense of sort-term success.

This is a problem for agile teams that think planning is unnecessary or who have a poor product owner. This leads to "incremental and iterative death marches".

. Agile can be hard on the product owner who has a lot of responsibility. They are often asked to provide real time requirements support, make feature and business decisions, define acceptance criteria, and be constantly available to answer questions. The product owner is often responsible for understanding and communicating the needs of the customer and the company. This role can become a bottleneck because it is unable to "feed" the development team fast enough.

Yes! Large projects need a PO for every Scrum team.

8. Agile is over-hyped, thus leading to unrealistic expectations. People are led to believe agile development will solve all their problems with minimal effort and experience disillusionment when it doesn't meet their expectations.

Backlash from the silver bullet myth. I often hear the phrase "yeah, we tried Scrum on a project and the project failed, so now we....".

Scrum doesn't insure success. At best it creates transparency so you can make better decisions. If you choose the wrong project or the wrong team for a project, Scrum won't insure success; it will point these facts out to you quickly. If you decide that the problems can't be fixed and the project must be canceled early, that's better than waiting until the end.

9. A variation on
The Blame Game can arise. A moderately successful agile development team usually will no longer be the bottleneck. Other departments can no longer blame development and this may give rise to a shift in political games.

This is absolutely true. Troubled development projects hide a lot of problems in the rest of the chain. Marketing, licensing, etc. can be caught flatfooted by an agile team that delivers at a far steadier pace.

10. Too much power can be granted to the product owner when it comes to steering the product. If they have an agenda, they can cause a lot of damage. Agile teams often seem to lack sufficient checks and balances.

Yes. A Product Owner who ignores the value of iteration can cause problems. You don't want to replace a plan on paper with a plan in someone's head. A Product Owner has to communicate their vision and react to the emerging game.

11. Agile is too programmer-centric leaving it unclear how to balance work across an organization. There is a need for better documentation and coaching for non-development participants.

This is the purpose of "Agile Game Development" and what I do.

Thanks to Jason Yip for pointing out the original post.

Wednesday, January 07, 2009

"Sorry Goose, but it's time to buzz a tower"

In 1990 I was a member of the team that was developing the avionics testbed for a experimental fighter jet called the YF-23. This work required me to live for almost a year at a McDonnell Douglas facility in St. Louis. Members of the team had gathered from all around the company to prepare the avionics for demonstration to the Air Force. We faced many imposing challenges; the various components of hardware and software had been separately developed and were resisting integration. The avionics were designed to survive destruction of up to half of their components and still perform their function. Unfortunately the actual hardware could hardly tolerate being installed. One key component, a fiber optic communication interface was so sensitive that 29 out of 30 initial boards produced failed before our final demonstration!

The team was led by a former F-14 pilot. He was an outstanding leader. He didn’t understand the details of how each of us did out jobs. He hadn’t written a line of code in his life. What he did very well was remove obstacles from our path.

We were guaranteed to see him every morning at the daily stand-up meeting. Scrum was largely unknown to the world in 1990, but apparently F-14 pilots knew how to have a stand-up meeting. Each of us, in turn, would describe our progress, what we were doing next and what problems we were having.

Our F-14 pilot-lead had this interesting habit that I have never forgotten: he used to trim his nails every day during this meeting. He focused on his nails, but you knew he was listening. I didn’t realize it at the time, but it forced me to speak to the group instead of to him. If the discussions got too involved, he would cut it off.

During one of the first daily meetings I reported that a McDonnell Douglas system administrator was not giving us access to a computer as he had promised a week earlier. It was cutting into our efforts to test the avionics. The administrator was being a jerk to us contractors.

As soon as I told the story, our lead’s head snapped up. With a steady glare he repeated what he heard me say. I verified that he had heard me right; the administrator was messing with the team. I thought I was in trouble, but he didn't say anything else about it.

Five minutes after the conclusion of the meeting, we heard our lead swearing at the top of his lungs at the administrator. They must have a class for F-14 pilots on the creative application of profanity. It was impressive to hear. It was even more impressive to realize that our pilot-lead had our back. He was our “wingman”. As Tom Cruise learned in “Top Gun”; you never leave your wingman.

We received access immediately and never had another problem with the administrator. It was a pivotal moment for the team. We started the day as a collection of contractors from around the country. By noon we were the team you didn’t mess with. Did it effect our work? You bet. We didn’t have any excuses not to solve our own problems with the dedication demonstrated by our lead.

Our lead wasn’t a Scrum Master, but he would have been a good one. Our daily meeting wasn’t a “Daily Scrum”, but it sure served the purpose of one. Scrum invented very little. It was derived from observing what worked and formed a framework to encompass these practices. While I don't teach Scrum Masters how to be as effective with their profanity as our lead was, I'm sure to let them know that their job is to be a wingman for their team.

Tuesday, January 06, 2009

Announcement: Certified Scrum Master course at GDC

I'll be giving a Certified Scrum Master for Video Game Development course/workshop in San Francisco the week of GDC (on Monday 3/23 and Tuesday 3/24). The course will focus on the game specific application of Scrum to game development including team exercises.

It's being held at the new Intercontinental Hotel, one block from the Moscone Center.

Monday 3/23 and Tuesday 3/24

Registration fee:
US $1500 per attendee

Registration and further details for the class can be found here.

No prior experience with Scrum is necessary. At the conclusion of the course, all attendees will be Certified Scrum Masters.

Thursday, January 01, 2009

Industry Broadcast Hits 50

Industry Broadcast, the podcast for the games industry, has hit the 50 article mark - about 16 hours of ear candy. The site carries a some of my articles and those by some of my favorite bloggers including Daniel Cook and Noel Llopis.