There is a great response on Quora for the question Engineering Management: Why are software development task estimations regularly off by a factor of 2-3? The top answer is right on the mark.

The main reason for this problem is that software development is not an engineering discipline. The idea that software could be engineered originated from the 1968 & 1969 NATO “Software Engineering” conferences.

“What software people must do is to adopt the same engineering practices that are successfully deployed by us hardware engineers” – Bell Labs 1968

Over 40 years later this misconception still persists, but it is a misconception that an increasing number of people within the software development community are trying to dispel.

“The Agile Movement is the end-game of a set of approaches starting in the late 1970’s that asserts business and software product development is a creative, open and intensely collaborative process not an engineered process.” – Rob Thomsett 2011

There will always be difficulties encountered that throw software development estimates completely off. This is why I hate estimates (Waterfall or Scrum based) and love Lean’s Cadence. From Mary Poppendieck:

A regular cadence, or “heartbeat,” establishes the capability of a team to reliably deliver working software at a dependable velocity. An organization that delivers at a regular cadence has established its process capability and can easily measure its capacity.

It is with this measured dependable velocity that lean software teams can accurately and reliably predict and plan their projects.