Unknown and Unknowable Data
From Out of the Crisis, page 121:
the most important figures that one needs for management are unknown or unknowable (Lloyd S. Nelson, director of statistical methods for the Nashua corporation), but successful management must nevertheless take account of them.
We need to manage systems even though we cannot collect data that would be extremely valuable if it were knowable and available. One of Dr. Deming’s list of 7 deadly diseases captures this point:
Management by use only of visible figures, with little or no consideration of figures that are unknown or unknowable.
Ironically, people often quote Dr. Deming as saying something along the lines of: “you can’t management what you can’t measure.” Not only did he not believe this, he thought managing as though it were true was a deadly disease of management.
Martin Fowler wrote a nice article on this topic within the context of measuring software development productivity, Cannot Measure Productivity
So not just is business value hard to measure, there’s a time lag too. So maybe you can’t measure the productivity of a team until a few years after a release of the software they were building.
I can see why measuring productivity is so seductive. If we could do it we could assess software much more easily and objectively than we can now. But false measures only make things worse. This is somewhere I think we have to admit to our ignorance.
Using data to aid in improvement efforts is extremely valuable. However, it is easy to be led astray while focusing on numbers without a solid understanding of concepts such as: variation, appreciation for a system, the proxy nature of data, theory of knowledge, etc..
Related: Manage what you can’t measure – The Idea of Performance Rating to Capture Merit is Alluring – Is the Result Due to Mathematical Probability or Individual Merit? – How to Manage What You Can’t Measure