The Planning Fairy Tale

Software project management is full of “Let’s pretend.” Let’s pretend we can write a full schedule before we know the requirements; let’s pretend we can estimate how long it will take to solve this unsolved problem; let’s pretend we can predict schedules to the hour or half day, two years in advance. Here’s how to spot a “Let’s pretend” schedule and what to do when you find one.

When we know we don’t know the final requirements, predictive schedules are a game of let’s pretend.

The start of a software project is an exciting time. There are new things to learn, new challenges, and the excitement of feeling, “This time we’re going to do things right!”

Then, shortly after the kick-off meeting, the project manager holds a planning meeting. Invariably, it seems, this starts with management platitudes. “This is just a stake in the ground.” Or “No one takes this too seriously; it’s too early.” But, they add ominously, “You’ve got to manage to something.” And the inevitable, “Management needs a date.”

So ideas are brainstormed, times are estimated, and someone – usually, for his sins, the project manager – is assigned to push those estimates into Microsoft Project, producing a pretty Gantt chart and an estimated delivery date.

Now, let’s not be too hard on project managers. Those management platitudes, all of them, are actually true. That initial plan really is just a stake in the ground, and in their heart of hearts, no one takes them very seriously. No one really thinks you can predict what you’ll be doing on June 14, 2014, no matter what the Gantt chart says. And you do need to manage to something; you’ve got to have some idea of what Melvin the programmer has to do next, and when to start hiring developers for the mobile application to be added in the second iteration.

Most of all, the project manager is absolutely right that Management needs a date. In fact, if you could peer directly into their pointy-haired hearts, management has a date, based on things like time to market, hardware deliveries, and regulatory changes. If the date that comes out of Microsoft Project is too much past the date they have in their hearts, there will be pushback. Anything from subtle pressures to flat-out threats.

In any case, people come out of the planning meeting with their initial enthusiasm quenched in the certainty that what was just a stake in the ground on Friday afternoon will be The Plan of Record on Monday morning. And they foresee a significant part of their future will include heroics to meet a schedule date, and endless negotiations to change The Plan as reality impinges upon it. The Plan of Record, the development team is sure, will run into the Two Ineluctable Facts of Project Planning:

  1. If you don’t know what you’re going to build, you can’t know how long it will take to build it.
  2. You only really know what you’re going to build when you finish it.

The planning exercise is been, in reality, a sort of undercover, more or less informal feasibility study to answer the question, “Do we think we can build something like this in something like the time available?”

However, because management needs a date, and you’ve got to manage to something, everyone leaves the planning meeting having made a tacit agreement: Let’s pretend this fairy-tale Plan of Record is real.

Now, I should say that once, in the beginning of my career, I worked on a project that actually went smoothly from start to finish. It made all its milestones, it delivered on time and under budget, and it completed a planned 30-day acceptance test in less than a week.

What’s more, this project started out perceived as very risky. The general opinion was that the responsible executive’s office should be fitted for foam-rubber wallpaper. For various reasons, most of them political, the contract had some interesting constraints:

  • It was denominated in a foreign currency, and my company accepted all the currency-exchange risk.
  • It was absolutely firm fixed price, no kidding and no exceptions.
  • The contract was written so that a very large part of the total payment would only be made after acceptance.

In general, a company like the one I worked for would run screaming from such a contract. The risks were too great. But somehow, the contract was signed.

In fact, the project was a success, risks notwithstanding. I’d maintain the project was actually a success because of these constraints, not in spite of them.

Due to the constraints, the contract had other interesting features. There was an initial study phase, also fixed cost and fee, to develop two things: a user’s manual for the system, and an acceptance test. The acceptance test consisted of about 300 pages of clear, Yes or No tests: “The operator does this, and that happens. Did it?” For the system to be accepted – meaning my company got paid – 100% of these tests had to pass.

This acceptance test was then written into the contract. There was no change mechanism, no engineering control board. Any changes beyond trivial errors would require actually renegotiating the whole contract.

In other words, what the constraints had done was force us to eluct the Two Ineluctable Facts: We could predict how long it would take, because we did know exactly what we were going to build.

Of course, I then spent a year with the customer making the changes they hadn’t been able to request. We had elucted the ineluctable artificially; the requirements had changed, the system we were building wasn’t really satisfactory, and we’d held off the changes only by building up a big backlog of development work.

So what can we do to keep our planning from being a fairy tale? We need to accept some basic facts:

  • The requirements will change.
  • The product will never be perfect.
  • We eventually need to release it anyway.

Taken together, these mean that all schedules are, in reality, time-boxed schedules. The initial planning exercise isn’t useless, but the reality is that it’s never anything more than a feasibility study. It’s when we ennoble it as the Plan of Record and expect to follow it faithfully that we get in trouble.

Since this is the reality, we can and should adapt our development process to changing requirements. There is a way to do this, and it’s actually been known since (at least) 1971, when Harlan Mills wrote about “Top Down Programming in Large Systems.” It’s called incremental development: “Build a little, test a little.” With every increment, you have a working system, and that system focuses everyone’s creativity toward bringing that system closer to the target. Every new increment is another working partial system that is closer to the imagined ideal system. When the time boxes run out, you have a working system. It may not be a perfect system, but then it never will be the perfect system.

Of course, what I’m describing is what we’ve come to call an “Agile process.” What we should call it is a realistic process. It’s a process that accepts and works with the way the real world works.

Trying to make the Plan of Record approach work is no more than a game of let’s pretend.

Related Article:

The New School of Systems Development: Simple, Dynamic, and Agile