Thursday, 22 January 2009

Measure for measure (why we're doing it wrong)

In this post I will outline that the way in which we measure productivity in software development is still flawed - even in agile.

One tenet of Lean is that the core activity of production is to deliver value to your customer. A second tenet of Lean is that something is not finished until the customer receives it. Only then do you record it on your balance sheet. Combining the two is a simple statement: the purpose is to deliver value to the customer; it is not delivered until the customer receives it.

Everything else is unimportant. Anything that is not contributing to the first tenet is waste (muda) anything that is produced but isn't delivered to the customer is inventory and inventory is also waste.

The relationship between productivity and value in Lean is fundamental. If something which is produced yet not delivered doesn't have any value then we have not produced. Regardless of how many hours worked, how many items output, how much blood, sweat and tears, if all you have created is inventory then you have as good as done nothing.

Measuring productivity has been a holy grail of software development. It sits at the heart of many of Agile methodologies. Thanks to essays such as The Mythical Man Month and with tools such as estimation, iterations, velocity and burn down charts Agile has moved us beyond measuring productivity with the highly flawed method of counting hours worked. Instead it breaks projects into units of functionality (stories) which are given sizes (points) relative to their complexity/effort. Productivity is then measured by totaling up the measure of effort once the functionality is completed and 'signed off'.

Using velocity to measure progress and productivity has resulted in a revolution in thinking. Using velocity teams are better placed to manage and schedule workload (scope) and much more able to accurately report what has been done and what is left to do. The result has been much more predictable and achievable project timelines.

However there is a disconnect between the customer (and user) and the producing team. The producing team is measuring its productivity based on how much effort is realized into working software. The customers success will be ultimately measured (whether they are aware of it or not) on how much value they have delivered (note the tense: not how much they potentially will/may deliver but HAVE delivered).

The result of this disconnect is that teams responsible for production are working within a model which is not based on reality. The real value of what they are producing is nowhere to be seen in any of the measurements or reports output by the team. The result is their measure of success being out of line with that of the customers. The team ends up working to its goal to maximize output of functionality but not to producing value.

Teams need to change the way they measure productivity to bring it inline with the way the customer is measured. To achieve this is simple: measure velocity in terms of value, not effort. And by keeping with the Lean tenet to only record value when it is delivered to the end user.


Torbjörn Gyllebring said...

Really nice written post. But, in Scrum the definition of done is supposed to be "potentially shipable software" or if I recall correctly we're encouraged, dare I say challanged to push the definition even further to include "in the hands of real users" and this should be what agile, or at least Scrum teams measure their progress against.

Anything less than "used" should be handled as an impediment so I really don't think agile and lean are diffrent here, just that the lean folks are stressing the point of that it isn't value unless someone is using it a bit more than most agile software folks are used to.

The ultimate goal should always be to put valuable software in the hands of customers, if we ever forget that I think we're heading down a quite dangerous path.

Gil Zilberfeld said...

Nice post.
I think of velocity, not as a measure of productivity, rather as a tool for estimation. With velocity, we can answer the question (loosely) "When will it be done".
If you put the team accountable to a different velocity than what naturally their velocity is, you're doing a disservice to the team and management. If you don't like the velocity (as a manager) add or remove people and impediments.

You can't set a benchmark on productivity. Velocity is a remote proxy, yet still a proxy. Although it maybe the only proxy we have, so people can misuse it as you write.

Gil Zilberfeld

Peter Gillard-Moss said...

Thank you both for your comments. I don't at all disagree with either points.

Torbjörn: By only measuring value when something is in use we improve the process by making it explicit about the fact that this is our goal. Achieving points before use is reward for something not gained (it is almost fraud)

Gil: I agree that we need a tool for estimation. I will try and tackle this in a future post and demonstrate what I see as the relation between the two.

arneahl said...

Interesting post. It made me think, and that is a positive measure to me.

A short summary of my thinking: To me measuring productivity and measuring value are two different things. Yes, I want value, and lots of it. That does not mean that I am willing to give up productivity.
Looking at it from another angle, I want productivity, but productivity alone does not give me value. It might, but not necessarily.
I use velocity to measure productivity, and to support my estimation and planning. I strongly believe that is needed. In order to make sure I also get value from the productive team, I need to measure value. To me that is up to someone else than the team. E.g. a Product Owner/Agile project manager/product manager.
It is the responsibility of that role to make sure we work on the right thing (valuable). It is the responsibility of the team to make sure the value is obtainable. I agree: I don't like unrealised value. To me that is waste.
So what to do? Let the product owner prioritize based on value/cost, let the team strive for high and even productivity. Let the product owner and the team collaborate around how to realise the value as soon as possible.

About Me

My photo
West Malling, Kent, United Kingdom
I am a ThoughtWorker and general Memeologist living in the UK. I have worked in IT since 2000 on many projects from public facing websites in media and e-commerce to rich-client banking applications and corporate intranets. I am passionate and committed to making IT a better world.