I recently had a conversation over breakfast with a number of senior and influential people on the subject of measurement and reporting. The talk centred around the question of how organisations can report on their agile projects effectively. Or, more importantly, communicate about them.
Why is agile reporting so special? Well, clearly an iterative approach to development provides an opportunity for measurement at regular intervals. And traditional reporting of percentage progress against set milestones is neither appropriate nor accurate. Surely this is easy, I hear you say. We have story points on all our stories and we have burndown charts that show how much we deliver every timebox. Easy!
Well, the people in the room didn't see it that way. You see, the CFO and the business leaders don't know what a story point is and don't care either. "Burndown? Sounds dangerous to me."
What they care about are what they are going to get, when they are going to get it and how much it will cost.
Apart from being inappropriate for agile projects, the problem with traditional progress reporting is that it is almost entirely subjective and biased. Some project managers are strongly incentivised to report good news, as the IT department cannot be seen to be doing a less-than-perfect job. Some companies have people whose job it is to turn red status reports into green ones. Not by changing what's going on, just by… aggregating and filtering the reporting before it gets to the upper echelons. Not only ineffective, this is downright destructive.
So we need some objective reporting, based on real data, right? Yes and no. While agile methods provide an opportunity for regular objective – and subjective – measurement, there is still the need to have a conversation with the senior stakeholders. They do not want a lot of numbers and graphs that they don't understand. As I mentioned earlier, their needs are simpler.
In some cases, this problem is exacerbated when a contract is involved. Despite the agile manifesto being ten years old, many organisations it appears, are still struggling with client-supplier contracts that protect both parties involved while still permitting the flexibility and collaboration that is needed for agile projects to succeed. How do you report to your client how much of the scope you have delivered when the user stories keep changing?
So we really need to balance the need for truth, objectivity and the avoidance of 'spin' with the need for clear and simple status updates in a form that everyone can understand.
Perhaps the answer lies in the concept of "Feature points" or "Epic points". It relies on the fact that requirements only change at the detail level; at the highest levels they remain stable throughout the project. If you state the vision for the project in a single sentence, then decompose that into the next layer of requirements you will probably have anywhere from a handful to a couple of dozen high-level Epics. Each of these is likely to be a Must Have. It's when they get broken down further and the detail emerges that lower priority requirements emerge. If our measurement is based on data at the detail level, it changes too much to be meaningful. And we do NOT want to advocate fixing those requirements up front just so we can measure how we are progressing.
So let's take those high-level Epics and somehow measure them – in "epic points" if you like, and report on those as and when they are delivered. This way, we measure and report on what the business care about, objectively, and without spin. Either it has been delivered or it hasn't. The key is in defining those high-level epics.
There needs to be enough of them to make regular reporting feasible and allow extrapolation of progress across the entire scope. If it took one month to deliver 20 epic points, then it should take another four to deliver the remaining 80 epic points that constitute the rest of the project. And we can flex the features at the detail level to accomplish this.
During the summing up of the session, one chap mentioned that this was cutting-edge stuff and very few, if any, organisations are doing it well. The key, as with so many other things, lies in communication. About requirements, priorities, estimates and not least of all, risks.
I would be interested to hear from anyone who has got this stuff right at the organisational level and is willing to share their experiences. Or from anyone who has anything else to contribute to the topic. Continue reading