The Planning Fairy Tale

on April 8, 2013

Software project management is full of “Let’s pretend.” Let’s pretend we can write a full schedule before we know the requirements; let’s pretend we can estimate how long it will take to solve this unsolved problem; let’s pretend we can predict schedules to the hour or half day, two years in advance. Here’s how to spot a “Let’s pretend” schedule and what to do when you find one.

When we know we don’t know the final requirements, predictive schedules are a game of let’s pretend.

The start of a software project is an exciting time. There are new things to learn, new challenges, and the excitement of feeling, “This time we’re going to do things right!”

Then, shortly after the kick-off meeting, the project manager holds a planning meeting. Invariably, it seems, this starts with management platitudes. “This is just a stake in the ground.” Or “No one takes this too seriously; it’s too early.” But, they add ominously, “You’ve got to manage to something.” And the inevitable, “Management needs a date.”

So ideas are brainstormed, times are estimated, and someone – usually, for his sins, the project manager – is assigned to push those estimates into Microsoft Project, producing a pretty Gantt chart and an estimated delivery date.

Now, let’s not be too hard on project managers. Those management platitudes, all of them, are actually true. That initial plan really is just a stake in the ground, and in their heart of hearts, no one takes them very seriously. No one really thinks you can predict what you’ll be doing on June 14, 2014, no matter what the Gantt chart says. And you do need to manage to something; you’ve got to have some idea of what Melvin the programmer has to do next, and when to start hiring developers for the mobile application to be added in the second iteration.

Related: Traditional App Delivery is Broken, And How you Can Fix It Now

Most of all, the project manager is absolutely right that Management needs a date. In fact, if you could peer directly into their pointy-haired hearts, management has a date, based on things like time to market, hardware deliveries, and regulatory changes. If the date that comes out of Microsoft Project is too much past the date they have in their hearts, there will be pushback. Anything from subtle pressures to flat-out threats.

In any case, people come out of the planning meeting with their initial enthusiasm quenched in the certainty that what was just a stake in the ground on Friday afternoon will be The Plan of Record on Monday morning. And they foresee a significant part of their future will include heroics to meet a schedule date, and endless negotiations to change The Plan as reality impinges upon it. The Plan of Record, the development team is sure, will run into the Two Ineluctable Facts of Project Planning:

1. If you don’t know what you’re going to build, you can’t know how long it will take to build it.

2. You only really know what you’re going to build when you finish it.

The planning exercise is been, in reality, a sort of undercover, more or less informal feasibility study to answer the question, “Do we think we can build something like this in something like the time available?”

However, because management needs a date, and you’ve got to manage to something, everyone leaves the planning meeting having made a tacit agreement: Let’s pretend this fairy-tale Plan of Record is real.

Now, I should say that once, in the beginning of my career, I worked on a project that actually went smoothly from start to finish. It made all its milestones, it delivered on time and under budget, and it completed a planned 30-day acceptance test in less than a week.

What’s more, this project started out perceived as very risky. The general opinion was that the responsible executive’s office should be fitted for foam-rubber wallpaper. For various reasons, most of them political, the contract had some interesting constraints:

  • It was denominated in a foreign currency, and my company accepted all the currency-exchange risk.
  • It was absolutely firm fixed price, no kidding and no exceptions.
  • The contract was written so that a very large part of the total payment would only be made after acceptance.

In general, a company like the one I worked for would run screaming from such a contract. The risks were too great. But somehow, the contract was signed.

In fact, the project was a success, risks notwithstanding. I’d maintain the project was actually a success because of these constraints, not in spite of them.

Due to the constraints, the contract had other interesting features. There was an initial study phase, also fixed cost and fee, to develop two things: a user’s manual for the system, and an acceptance test. The acceptance test consisted of about 300 pages of clear, Yes or No tests: “The operator does this, and that happens. Did it?” For the system to be accepted – meaning my company got paid – 100% of these tests had to pass.

This acceptance test was then written into the contract. There was no change mechanism, no engineering control board. Any changes beyond trivial errors would require actually renegotiating the whole contract.

In other words, what the constraints had done was force us to eluct the Two Ineluctable Facts: We could predict how long it would take, because we did know exactly what we were going to build.

Of course, I then spent a year with the customer making the changes they hadn’t been able to request. We had elucted the ineluctable artificially; the requirements had changed, the system we were building wasn’t really satisfactory, and we’d held off the changes only by building up a big backlog of development work.

So what can we do to keep our planning from being a fairy tale? We need to accept some basic facts:

  • The requirements will change.
  • The product will never be perfect.
  • We eventually need to release it anyway.

Taken together, these mean that all schedules are, in reality, time-boxed schedules. The initial planning exercise isn’t useless, but the reality is that it’s never anything more than a feasibility study. It’s when we ennoble it as the Plan of Record and expect to follow it faithfully that we get in trouble.

Since this is the reality, we can and should adapt our development process to changing requirements. There is a way to do this, and it’s actually been known since (at least) 1971, when Harlan Mills wrote about “Top Down Programming in Large Systems.” It’s called incremental development: “Build a little, test a little.” With every increment, you have a working system, and that system focuses everyone’s creativity toward bringing that system closer to the target. Every new increment is another working partial system that is closer to the imagined ideal system. When the time boxes run out, you have a working system. It may not be a perfect system, but then it never will be the perfect system.

Of course, what I’m describing is what we’ve come to call an “Agile process.” What we should call it is a realistic process. It’s a process that accepts and works with the way the real world works.

Trying to make the Plan of Record approach work is no more than a game of let’s pretend.

Related Article:

The New School of Systems Development: Simple, Dynamic, and Agile

Evolving requirements, tight deadlines, small teams? Develop enterprise applications faster then ever and deliver results in days instead of months. Watch 2 minute video and get started for free at

  • Mike K

    “When we know we don’t know the final requirements, predictive schedules are a game of let’s pretend.”

    That may be true, but even if you do know every last final requirement (and that’s rare enough but not entirely impossible) predictive schedules are still mostly works of fiction. You might have removed one layer of uncertainty, but you still have plenty of other layers of uncertainty to deal with, especially if you’re dealing with creating something new.

    The only time you really can predict the finish date of a fixed scope project with accuracy is when you are repeating a relatively trivial exercise you’ve done many times before (e.g. add an extra field), but even then beware, many developers (well humans) are too optimistic and see more similarities and fewer differences than actually present.

  • Yes, absolutely. I’m pretty much convinced, as I said, that all schedules are really time-boxed schedules. The one special case I described, if done as a real time-boxed schdule with iterations, could have included a fair bit of new functionality and corrected some changes; it was just too risky to allow.

  • Sara Campos

    Nice article. I experienced the same problems as Architect for custom development. Eventually, we instituted a “Requirements” phase for all projects. It involved me, not a business requirements person, to understand everything and design the solution. Once we received sign-off on my requirements/design specification we would provide a timeline for delivery. The design phase period would take between 2 and 6 weeks depending on back and forth changes. The nice thing is we generally got a handle on timelines after that change.

    It was hard to keep that phase of the process since management and others liked to promise stuff with hard deadlines. But for the projects that we could enforce this method, they went well.

    The problem still exists in the industry, and I doubt it will ever go away; just human nature. Who doesn’t want to predict the future?

  • chasrmartin

    Thanks, Sara. I really think the key is keeping the cycle short enough that you can deliver little pieces quickly. That, and making sure that the people who are decided *what* they want are talking about it with those of us who are supposed to be building it.

  • Mark C

    I’ve been around since punch cards too, and in my opinion the biggest factor by far is that nobody — most particularly management — wants to admit what they know perfectly well to be true: that’s it’s going to take longer than they want it to take. It’s not primarily a matter of unknown requirements or missing information, though those are definitely complicating issues. Everyone involved knows perfectly well that the schedule is unrealistic.

    But management like it that way. In fact they push and push for lower estimates. They believe that if the schedule were realistic, people would slack off until it’s too late. They want people worried up front. Plus they have higher levels of management that they are lying to for assorted reasons, mainly because executives demand that it be done by a certain time, and middle management is too wimpy to tell them the truth.

    Everyone — executives, middle managers, engineers — is lying to everyone else and everyone knows it. We’re all participants in a giant fiction that we agree to because it’s easier to do that than to face the truth. But the primary blame in my opinion lies with middle management. Executives beyond a certain level lack the technical knowledge to evaluate schedules. It’s the first and second-level managers and directors who know what they are asking is unrealistic, but do it anyway.