The W. Edwards Deming Institute Blog

Data is Important and You Must Confirm What the Data Actually Says

by    

“Change the rule and you will get a new number.”

Attributed to W. Edwards Deming. Direct quote from The New Economics “If you change the rule for counting people, you come up with a new number.”

Dr. Deming emphasized the importance of understanding what the data actually meant (and how easily we can be mislead when looking at data). Without understanding the operational definition for the data you take a great risk that you make mistaken guesses about what the data means.

“An operational definition is a procedure agreed upon for translation of concept into measurement of some kind.”

W. Edwards Deming, The New Economics, page 105.

Another quote attributed to W. Edwards Deming that is important for those using data to consider:

Nobody should try to use data unless he has collected data.

This echoes the idea of going to the gemba. If you have not collected data you often fail to understand the judgement calls that go into assigning and recording a result from real world conditions. If you don’t have operational definitions (which is a very common condition) there is a significant risk that data doesn’t provide a decent view of reality. Without an appreciation for the gemba, where the data was collected, it is easy to be mislead by the data.

The most common waste of effort in examining data is reacting to the expected variation of a system as if it is something special. We have discussed this in many previous posts (for example, We Need to Understand Variation to Manage Effectively). After that I think there is a good chance failure to appreciate what the data is (and what it is not) telling us (based on mistaken assumptions about what the operational definitions were – this happens implicitly not explicitly) is the next biggest source of wasted effort.

To counter the problem (of data that has introduced variation beyond the system itself due to the method of data collection), make sure you understanding what the operational definition used to collect the data was. If you do not have operational definitions for data provided in your organization, as you often will not, you need to figure out if the data is worth trusting (possibly by visiting the gemba) or whether you need to establish operational definitions and make sure new data is collected using the operational definitions.

Often the variation in the collection of data will hamper your ability to use PDSA. If you practiced PDSA as you should, this wouldn’t be a problem, because you would make sure data had operational definitions. But often data used in PDSA effort isn’t collected with operational definitions (so, it is in understanding this reality that the comments about using such data becomes relevant).

Without reliable data, the conclusions that will be drawn from results ads a great deal of potential for error to your effort. The potential for data discrepancies increases the larger systems get. Differences in performance between locations can often be due, not to actual differences in the processes themselves, but in the decisions made in collecting the data.

Related: Where There is Fear You Do Not Get Honest FiguresData are not taken for museum purposes; they are taken as a basis for doing somethingDangers of Forgetting the Proxy Nature of Data


Categorised as: data, Dr. Deming, understanding variation


5 Comments

  1. […] data to help guide improvement efforts. But it is critical to know what the data does and does not tell you. There are many problems created when people attempt to use data but don’t understand what […]

  2. […] One of the four areas of Deming’s management system is “understanding variation.” The core principle underlying that concept is using data to improve while understanding what data is and is not telling you. […]

  3. […] Using data to understand your processes and improve them is very useful.
    But using data often results in unintended consequences. If you don’t […]

  4. […] mentioned. And in seeing that they can understand that just because you see variation in data that doesn’t mean you can identify causes – often there is no cause it is just variation (also see the Dangers of Forgetting the Proxy Nature of […]

  5. […] logical extension of this is that in order to understand data / information we must have an operational definition of the context in which they were obtained, that includes the […]

Leave a Reply

Your email address will not be published. Required fields are marked *