Aterny Blog

Can we believe the Chaos report?

16 Jul 2013

The Standish Group have been publishing the Chaos report into the state of software development annually since at least 1995 and the figures have been reproduced all over the place, especially when trying to explain why Agile is a good idea. If they are to be believed, the software development industry is in serious crisis, with only 32% of projects completing successfully (in 2009).

But some people have been questioning the figures. Ask yourself – How many failed projects are you aware of? I haven’t seen too many, to be honest.

Interestingly, the Standish Group have never published their data, nor the names of companies that completed their surveys. Which raises questions about the integrity of the surveys. But more to the point, the definitions of what constitute a failed or challenged project is seriously open to question. The University of Amsterdam and the IEEE highlight the problems with the way the Chaos study has been conducted and challenges the results as being unrealistic.

“…Standish defines a project as a success based on how well it did with respect to its original estimates of cost, time, and functionality”. In other words, they are judging not the projects themselves, but merely the estimation of those projects. And that is something entirely different. That project success can be meaningfully defined as the accuracy of estimation is clearly nonsense.

Scott Ambler regularly conducts surveys on a host of IT topics, especially but not limited to Agile. According to these surveys, in 2011, traditional projects were perceived to be successful about 50% of the time, Agile or iterative projects about 70%. Yet, in 2010, agile projects were perceived to be successful just 55% of the time, challenged 35% of the time, with 10% failing. Interestingly, 33% said that no agile projects had failed.

While these figures are better than those published by Standish, I think there is some way to go before we properly understand how successful projects are, especially in an agile context. Are we all measuring the same thing? In my recent experience, I can think of only two project failures in the last couple of years – less than 5 percent! One was stopped part way through, the other was completed but had to be ‘switched off’ afterwards.

Perhaps we need a new definition for success, failure and what constitutes compromised, that can be applied to all IT projects, regardless of Method. Then we can realistically assess how successful Agile methods are.

What does success mean in your organisation and what are your success rates? Answers in the comments box below please.

Last Modified: -/-
Chaos failure Standish Success survey

No Comments Yet...

Leave a reply

Your email address will not be published.