Why is estimating initiatives so stressful?

You already know the estimate is way off a few weeks after starting the project. So much effort went into estimating and then into the trash bin. But why? And how to do it better next time?

2023, May 15

First, we need to understand what is an estimate. We are at a certain point in time, looking ahead, trying to make an educated guess about the future. That’s it—a guess.

So, before we dive into estimation problems and how to estimate better, let’s get one thing straight:

PROJECTS DO NOT GET LATE. They take as long as they need to get done. Our job is to define what is “done” and review the estimates frequently.

If we had a time machine that allowed us to travel into the future, look how long the project would take to get done, that’s the number we would choose for our forecast. However, we don’t have a time machine. Therefore, it’s your guess that is wrong. It’s as simple as that.

Until you stop saying things like “the project will go over the deadline” or “the project is late,” etc., you won’t be able to look at the real problem and fix it: your estimate.

The Challenge

Voros, J. (2019). Big History and Anticipation. In: Poli, R. (eds) Handbook of Anticipation. Springer, Cham. https://doi.org/10.1007/978-3-319-91554-8_95, which was adapted from Hancock, T., & Bezold, C. (1994). Possible futures, preferable futures. Health Forum Journal, 37 (2), 23–29.

Estimation is a guess with partial knowledge, but we are biased to see what is there, not what is missing. We often estimate with the information made available to us. Still, we neglect to look at what information we don’t have (below, you will find the link to a post where I list some essential questions you should ask when providing high-level estimates).

The image above is called “Futures Cone.” At the beginning of the initiative, you are projecting a future based on what you think is preferable and trying to find the intersection of what is probable.

Some people like to say estimation is just predicting the future. As if it wasn’t hard enough, I like to say it’s worse because you are trying to predict a future that you *think* you know what it is.

One thing that people OFTEN lack is the perception that the preferable future is unknown. As you start seeing things get completed, you learn from them. You don’t just learn about the actual productivity of the team and the complexity of the work at hand, but you also get feedback from other people regarding the scope.

Ideas and suggestions come in; sometimes, implementing them is better than fully releasing your original intent. Changing the scope is changing what is preferable, and you can’t know how much it will change when it’s too early.

That’s why you should wait until closer to the end before making specific commitments and always prioritize the scope. The MoSCoW method is excellent for this purpose, but there are other techniques too. You can read about them here:

Good x Bad Estimate

Let’s define our target. What is a good estimate?

• It’s based on the best information available at the time.
• It’s a range (or a probability distribution), not a number. If your estimate is a number, it’s a target. It must account for uncertainty.
• It’s less precise the earlier you are in your initiative execution (which implies that you should monitor and refine the progress over time). Imprecision is welcome.
• It’s recent. If the project hasn’t started, consider that people move away; new people join, and the process and technology change. If the project has started, the more you have done, the more is known. What has passed is not subject to estimation anymore. It’s set. You can estimate better the remaining work. And remember, the shorter the project is, the more often you should be monitoring and re-estimating to take corrective actions sooner.
• Either the scope is fixed with the completion date estimated, or the delivery date is set with the scope estimated. Don’t EVER start from a deadline and choose what should be done by when “to make the scope fit.” Determine whether you are supposed to be estimating the schedule or figuring out how to hit a target. If you need to hit a target, there is some flexibility by cutting scope (the most straightforward way), augmenting the team, improving the process or the tools the development teams will use, delegating some work to other teams or organizations, etc. Still, there’s a limit to what can be done.

• The schedule is better when overestimated (vs. underestimated), and the scope is better underestimated (vs. overestimated).

McConnell, S. (2006). Software Estimation: Demystifying the Black Art. apud Charles R. Symons. 1991. Software sizing and estimating: Mk II FPA (Function Point Analysis). John Wiley & Sons, Inc., USA.

When estimating the schedule, the impact of overestimation is simply that your project may expand because there’s more time (Parkinson’s Law), inviting scope creep. However, the estimation limits it, and the project can be cut shorter by closely monitoring the scope.

Underestimating the schedule is much worse, with much higher penalties. Quality decreases, resources needed may no longer be available, dependencies are impacted, there may be regulatory/financial penalties, costs associated with renegotiations and communication, potential impacts to image and expectations, etc.

An analogous situation happens with the scope. If you underestimated, you have a choice to release earlier or to add scope. If you overestimate, you will face the same issues with schedule underestimation.

Estimating is hard —  The Forces at Play

Adapted from Barry W. Boehm. 2002. Software engineering economics. Software pioneers: contributions to software engineering. Springer-Verlag, Berlin, Heidelberg, 641–686. The original concept is much older, from Bauman, C. (1958, April). Accuracy Considerations for Capital Cost Estimation. _Industrial & Engineering Chemistry, 50(4), 55A-58A. https://doi.org/10.1021/i650580a748_

This is the cone of uncertainty. I’ve adapted it from Barry Boehm, including the common biases that Engineers, Product Owners, and Executives often display. Unfortunately, it’s almost always on the underestimation side.

This phenomenon is called the “planning fallacy”: predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed, which leads us to Hofstadter’s Law (It always takes longer than you expect, even when you take into account Hofstadter’s Law.)

This may happen because we usually remember things as better than they were (rosy retrospection), which biases our estimation through anchoring.

These are biases that everyone is subject to. However, there are some key characteristics involving each of these roles.

Role Estimation Bias

Engineers tend to focus solely on their activities. Developers tend to focus on writing code and (sometimes) testing. They tend to forget pre-delivery activities, project initiation phases, requirements gathering, etc.

Project owners tend to believe that the more features and shiny new things go out of the door, the better they will be perceived, displaying a tendency to push for something to be done earlier by questioning the estimates. Because developers usually concede, they believe they were right in pushing back on an estimate that was already likely lower than it should have been.

Unfortunately, stating a different number doesn’t make the project go faster. In fact, it may make it go slower because of the many shortcuts developers will have to take in the form of technical debt that will exact its pound of flesh. Also, the added pressure will likely make them create more defects.

Executives sometimes are forced to demand deadlines due to external regulations. However, many deadlines are also wishful thinking. They are based on the desire to have something ready by a particular time because of external threats. Unfortunately, it doesn’t make it feasible. This is a point where it’s imperative to assess the required scope and the MVP.

Deadlines are often imposed early in the project, either upfront or upon receiving an estimation that doesn’t fit the expectations. In these situations, gathering as much information as possible upfront is essential to help support a better estimation. Here is a post I’ve written recently with questions that may be asked at very early stages and will help with better initial estimations.

The worst mistake at the organizational level

I could have made this a bullet somewhere in this post. I wouldn’t dare to, it deserves its own section.

There is one management practice that is sometimes employed as a good thing. In my professional experience, it creates the typical environment people run away from, obliterating the organizational culture and pitching people against each other. I’m talking about using Parkinson’s Law to get more done in less time.

Work expands to fill the time available for its completion.
 — Cyril Northcote Parkinson, British historian and author.

This is done by pushing for things to be done in shorter deadlines than possible, knowing they won’t be done in that time, just to prevent work expansion, at a cost that executives are often oblivious to.

This practice is poisonous for software development as it creates an unsustainable rhythm, leading to burnout, high turnover, and loss of knowledge and talent. It also causes software brittleness and productivity loss due to a lack of quality and excessive technical debt (which sometimes forces organizations to rebuild existing solutions as they need to be better for the new requirements). The toll, in the long run, is tremendous.

We have already discussed that there are better ways to deal with this issue, for instance, by closely monitoring the scope already delivered, comparing it to the original intent, and weighing the remainder scope’s cost-benefit. 

As previously discussed, having schedules underestimated is worse than having them overestimated. Having them often underestimated is many times worse than having them often overestimated.

Estimation Techniques

Different estimation techniques apply to projects depending on the organizational maturity level, the amount of information available, and the project progression. I have written an entire post with the most popular estimation techniques, so I’ll leave it down here:

Cheers!

 Back