The W. Edwards Deming Institute Blog

Unknown and Unknowable Data

by    

From Out of the Crisis, page 121:

the most important figures that one needs for management are unknown or unknowable (Lloyd S. Nelson, director of statistical methods for the Nashua corporation), but successful management must nevertheless take account of them.

We need to manage systems even though we cannot collect data that would be extremely valuable if it were knowable and available. One of Dr. Deming’s list of 7 deadly diseases captures this point:

Management by use only of visible figures, with little or no consideration of figures that are unknown or unknowable.

Ironically, people often quote Dr. Deming as saying something along the lines of: “you can’t management what you can’t measure.” Not only did he not believe this, he thought managing as though it were true was a deadly disease of management.

Martin Fowler wrote a nice article on this topic within the context of measuring software development productivity, Cannot Measure Productivity

So not just is business value hard to measure, there’s a time lag too. So maybe you can’t measure the productivity of a team until a few years after a release of the software they were building.

I can see why measuring productivity is so seductive. If we could do it we could assess software much more easily and objectively than we can now. But false measures only make things worse. This is somewhere I think we have to admit to our ignorance.

Using data to aid in improvement efforts is extremely valuable. However, it is easy to be led astray while focusing on numbers without a solid understanding of concepts such as: variation, appreciation for a system, the proxy nature of data, theory of knowledge, etc..

Related: Manage what you can’t measureThe Idea of Performance Rating to Capture Merit is AlluringIs the Result Due to Mathematical Probability or Individual Merit?How to Manage What You Can’t Measure


Categorised as: data


4 Comments

  1. I think it was Albert Einstein who said that “Not everything that can be counted counts and not everything that counts can be counted”. Deming’s aphorism is, of course, paradoxical and likely ironic (pace Frank Knight). Unknowable figures cannot be considered. This is not simply about ideas that we cannot operationalise. It is about concepts we don’t yet have. It’s not about considering the known factors that don’t apear on the process behaviour chart. It’s about realising simultaneoulsy that the common cause variation sets a lower bound on our ignorance of causes and that the special cause always lies in wait to ruin the best laid plans.

    I think that Daniel Kahneman has captured this idea best by identifying the fallacy of WYSIATI (What You See Is All There Is). There is no easy solution other than being vigilant for the surprising observation and always investigating.

    I think also that it’s wrong to intepret Deming as warning against being driven by measrements. Au contraire. I remember the case of a car company who had a brake squeal problem. It had defied analytical attack. They told their brake pad supplier to measure the resonant frequency of the brake pads and then work on reducing variation. It eliminated the problem. By working on reducing variation in something, perhaps marginally characteristic, the variation in the unknown and unknowable physics was reduced.

  2. Alastair Philp says:

    as i tweeted in June (see below) not sure it was Einstein who said “not everything that can be counted counts…”

    alastair philp (@ajvphilp)
    12/06/2013 20:39
    @ilseja1 @krisvanhaecht @peterdegadt re: not ev’thing that cn b counted… Einstein? see pic.twitter.com/amwtUdsPsC & p2 of http://www.rwjf.org/content/dam/farm/reports/reports/2013/rwjf406195

  3. John Hunter says:

    I find the quote investigator website quite good. Here is their post on the quote in question http://quoteinvestigator.com/2010/05/26/everything-counts-einstein/

  4. “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” – W. Edwards Deming

Leave a Reply

Your email address will not be published. Required fields are marked *