How to Use Data and Avoid Being Mislead by Data

One of the four areas of Deming’s management system is “understanding variation.” The core principle underlying that concept is using data to improve while understanding what data is and is not telling you.

The mistakes in interpreting data are very often related to mistaking natural variation in data as meaningful. Combining this with our brains ability to find patterns (even from random data) and confirmation bias this creates problems. Using data is very powerful but it is not enough, you need to use data properly (and pay attention to other important factors data can’t adequately account for).

Data can’t lie, but people can be mislead and they can even mislead themselves by misinterpreting data.

image of the cover of book: How Not to be Wrong

How Not to be Wrong is an excellent book by Jordan Eilenberg on how to use math to avoid making mistakes. A great deal of the book is about the dangers of mis-interpreting data and how to avoid being misled.

The book doesn’t discuss variation directly but discusses many ways to be misled by incorrect interpretations of data.

When a theory really has got your brain in its grip, contradictory evidence – even evidence you already know – sometimes becomes invisible.

This is obviously not a problem with math it is an issue of our psychology. And this point is very well understood by those familiar with Deming’s ideas. It directly ties to 2 of the other areas of Deming’s System of Profound Knowledge: psychology and theory of knowledge. The book focuses on how to use math to avoid making errors. In doing so he is wise enough to notice that one problem is we are often trapped by our brains even when we should know better.

This quote was written about a scientist missing fairly obvious evidence – likely because it just didn’t fit how he was viewing the issue. This is a very common pattern and something you need to attempt to break yourself out of. In my experience this is possible but it requires developing a habit of continually questioning what evidence supports your belief and trying to find evidence that undermines your belief. It seems to me scientists are better at doing this than most of us, but even they often fall into traps based on their beliefs and they fail to see lots of evidence that it is hard to understand how they missed it later.

We are all subject to similar psychological forces that lead us to accept what is comforting and reject what is troubling. Another tactic I find useful is to remember this and when you reject something that is troubling take a bit of extra time to see if it is sensible to do so, or if it is something you should investigate further. And to question if you are letting yourself accept weak evidence because it is comforting (perhaps due to confirmation bias).

Another tactic is to gain numeracy (literacy for numbers) and with that gain an ability to spot data fallacies that often lead people astray. The book provides several examples of traps to avoid.

Dividing one number by another is mere computation; figuring out what you should divide by what is mathematics.

I think this is a great quote; though I must admit I think of that as statistics not mathematics but he is in the mathematics department at the University of Wisconsin-Madison, not the statistics department so I can understand why he sees it this way. My father was in the statistics department there, which is probably why I see it the way I do.

Related: Statistical Techniques Allow Management to do a Better JobData are not taken for museum purposes; they are taken as a basis for doing something.We Must Remember the Proxy Nature of DataBigger Impact: 15 to 18 mpg or 50 to 100 mpg?

You may also like...

7 Responses

  1. Thanks for this post John. This is the part of Deming’s teaching that I often struggle with (understanding variation). I read Wheeler’s book Understanding Variation and it helped me with the concept, but I am challenged trying to apply it where I work. I often am not sure what to measure and if I do, I’m not sure how to measure it. Folks appreciate my burn down charts showing trends, but this is about the best I’ve been able to do. Do you have any recommendations on where I can look to help me get better at this?

  2. John Hunter says:

    Getting better at using data is a bit tricky, so struggling is fairly common.

    Probably the easiest thing to do is to stop reacting to normal variation (caused by the system) as if it were special. This isn’t super easy but it is the easiest step. And it does make a big difference even if it doesn’t seem very exciting.

    The idea of actually using data properly provides big benefit but it much trickier. Don Wheeler’s book is a great start. Making predictions and evaluating how those predictions turn out is also valuable. And in doing so often (though not always) it will also spur you to collect data. This process of predicting, figuring out what data to use to help do so (and to evaluate the results) and considering the result of the prediction and how well the predictions overall are working can help.

    You learn what data is often useful, you experiment with real data and real processes and you learn what needs to improve. If you are at least somewhat close to using data well then just doing it and learning from your experience is very useful. If you are really far off the experience might not help any 🙁

    The links in the post above I think provide some useful tips (and the links within the posts they link to…).


    “If you don’t have an answer for how you will use the data, once you get it, then you probably shouldn’t waste resources collecting it (and I find there is frequently no plan for using the results).”

    It isn’t uncommon that the measures you would like to have are just not realistically available or are hard to determine. How to get started in this is one of the tricker pieces in my experience. It is a place where consultants may be very helpful. If that isn’t an option another possibility is just to ask others at your workplace for ideas for metrics (there are issues with this and a big one is that many metrics will more likely to lead you astray than actually help).

    This can also be an area where seeing what others are using can be helpful. Because it is hard to think up what are great metric seeing what others are doing may provide insight. Of course, the ideas must be evaluate for whether they would work for you (even if they are right for others they may not be right for you – and many are not really right for others it is just a thing they measure and while they have associated it with good things maybe they are wrong (correlation but not causation]).

  1. December 19, 2016

    […] Most management systems would benefit from encouraging the challenging the accepted beliefs. They would benefit from encouraging the testing of beliefs and the examining of the results of those experiments. […]

  2. February 6, 2017

    […] By stratifying the data by potential causes that experts (on related processes and the problem you are investigating) think could be related you learn where to focus the investigation. […]

  3. May 30, 2017

    […] – Investment Risk Matters Most as Part of a Portfolio, Rather than in Isolation – How to Use Data and Avoid Being Mislead by Data – Paradigms: The Business of Discovering the Future by Joel Barker – Bad Weather is […]

  4. July 25, 2017

    […] understanding what the data does and does not say (understanding variation) […]

  5. April 18, 2018

    […] People must continually question what the data does and does not tell us. […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.