The W. Edwards Deming Institute Blog

Distorting the System, Distorting the Data or Improving the System


In Fourth Generation Management (I highly recommend this book, by the way), Brian Joiner provided an excellent summary of the options to get better “results” (as measured by the data used).

The options are to:

  • distort the system
  • distort the data
  • improve the system

Obviously we hope to improve the system; and we would like to have confirmation we have done so shown with data. Using data isn’t as simple as just including data in reports and in meetings. That data must be understood and we must have an appreciation for the dangers of misinterpreting data.

Without an understanding of variation we can often mistake variation with evidence (of failure or success). Also without understanding the proxy nature of data we can be led astray when that proxy association is changed over time.

If a measured value improves over time it needs to provide an accurate measure of what we meant to measure, for that to be an accurate indication of success.

For example, if the time to fix identified bugs declined that seems to be a good thing. But if we have achieved that improvement by distorting the data or distorting the system it is not good evidence for an improved system.

We could, for example, distort the system by stopping new development. We could reassign all software developers to fixing bugs and in thoroughly testing existing software. We could put in quick fixes that fixed the issue raised, but only do that. So instead of doing sensible things like asking why, to find solutions that may not only fix this bug but fix other potential bugs that haven’t been reported yet (or even fix processes that would reduce bugs from being introduced in the future) we just quickly fix the issue as narrowly as we can in order to keep the measured value (time to fix bugs) as low as possible.

Modifying the system is necessary to gain improvements. Sometimes a change could be seen as either a distortion or an improvement. Arguing over something that is close to distorting the system versus improving the system isn’t very useful. But often the distortions are fairly obviously causing harm to the overall system in order to achieve improvements in one measure.

Another way to improve the data is to change the system to adjust how things are measured. Don’t count a bug as being reported when the developer is told. Require that a form be completed and be found to have meet specific criteria before the clock starts counting. And also count a bug as fixed when the developer says it is. If the user finds it is not fixed force them to put in a new bug report and start counting the time all over again.

Distorting the data can take the form of outright lying. More often it will take the form of influencing the data through the judgements people make and in creating new criteria (maybe even including operational definitions, but often those are not even created).

The third method to getting better “results” is to improve the system. Making improvements that achieve better results (given the measure used) is often much more difficult than the first two options. This makes it very tempting to fall into using one of the first two methods.

As an organization matures in the use of evidence based management practices using the first two options becomes more difficult. Because the system is sophisticated enough to see attempts to claim improvement when actually only the limited data being looked at improved.

The problem is that many organization have not reached this level of an understanding of data and therefore often instances of distorting the data or the system outnumber instances of system improvement. Using data effectively requires understanding how measures capture the actual state of affairs and requires understanding the limitations of the specific measures we are using.

The desire is to improve not a number but the underlying system. When we are lucky the measures will nearly directly correspond to the system and distorting the data or the system is not going to work. But we are often not lucky. It is very easy to fall into the trap of focusing on the number without viewing the organization as a system or understanding variation or the proxy nature of data.

It is good to get in the habit of considering if the measured improvements are truly an indication of an improved system or merely the result of distorting the system or the data.

Related: Managing to Test Result Instead of Customer ValueBe Careful What You MeasureBrian Joiner Podcast on Management, Sustainability and the Health Care SystemProblems caused by management by targets or goalsWhere There is Fear You Do Not Get Honest FiguresThe Defect Black Market (extrinsic motivation damaging the system)

Categorised as: continual improvement, data, systems thinking


  1. […] Distorting the System, Distorting the Data or Improving the System by John Hunter – “It is good to get in the habit of considering if the measured improvements are truly an indication of an improved system or merely the result of distorting the system or the data.” […]

  2. Mark Graban says:

    Great stuff. I wish more people knew of and cited Prof. Joiner’s work. “Fourth Generation Management” has been very influential to me.

  3. The same process that delivered the defects delivered outcome or results that were not defective. Study of the defects alone presumes that the defects are necessarily produced by a special cause. Such a presumption may frequently be invalid…

  4. […] Often this is even encouraged (for several reasons including rapid management turnover), as it is fairly easy to manipulate data (or the system) to claim success – much easier than actually improving the long term health of the […]

  5. […] Related: Distorting the System, Distorting the Data or Improving the System […]

  6. […] is challenging. It is easier to make excuses for why improvement is not possible; just as it is easier to manipulate the data or manipulate the system compared to improving the system. But those organizations that achieve success were lead there by people that learned how to make […]

  7. Process thinking flows from viewing the organization as a system. Only by building a network of reliable processes is sustained excellence possible…

  8. W. Edwards Deming: “Management by numerical goal is an attempt to manage without knowledge of what to do, and in fact is usually management by fear.”

  9. […] brings us to response time targets. Putting aside the arguments that numerical targets are arbitrary and prone to causing dysfunctional behaviour*, a critical further point is that targets do not provide a method. Neither do they provide […]

  10. […] surprise to those who understand likely outcome of pressure to meet short term goals is a desire to manipulate the figure rather than improve underlying results). In Profit Beyond Measure by H. Thomas Johnson and Anders Broms, the authors provide many good […]

Leave a Reply

Your email address will not be published. Required fields are marked *