Thursday, May 6, 2010

Regression to the mean

Youngme Moon explains a pardox behind accountability:

The minute we choose to measure something, we are essentially choosing to aspire to it. A metric, in other words, creates a pointer in a particular direction. And once the pointer is created, it is only a matter of time before competitors herd in the direction of that pointer.

In the 1980s and 1990s, a number of prominent hospitals agreed to make public their mortality rates. The agreement was considered a breakthrough in hospital openness, promising to give patients the kind of insider view into hospital quality that they'd never been privy to before. If a hospital's mission is to heal, then what better way to audit the performance of a hospital than to track the ultimate measure of that healing ability?

What soon became evident, however, was that a hospital's mortality rate is a function of an elaborate host of factors - including the type of patients it admits, the amount of experimental research its doctors conduct, and the degree of care it provides - each of which can heavily conflate the intended meaning of the metric.

To put it more bluntly, it soon became evident that the easiest way for a hospital to improve its mortality rate would be to stop admitting the sickest patients. Yet if all hospitals were to do this, the overall effect on the medical system would be chilling: There would be fewer hospitals accepting the most challenging cases, experimenting with the riskiest treatments, becoming specialists in the most intractable disease areas. Hospitals wouldn't get better, they would simply become more like each other.

In recent years, the college ranking system has come under fire for precisely this reason - for dampening the likelihood that universities will experiment with models of pedagogy that may not reflect well in the metrics. The rankings have made it hazardous to be a noncomformist.

This, then, is the problem with uniform systems of measurement. The more entrenched a system of measurement, the more difficult it is for a deviant, an outlier, or even an experimenter to emerge. Another way to say this is to say that a competative metric, any competative metric, tends to bring out the herd in us. The dynamic can be likened to the observer effect in physics, only applied with too little foesight: The act of measurement changes the behavior of the thing being measured.
Many see test scores as a valid indicator of good schools, good teaching and good learning. And they wish to make the data public. Like the hospital example, this is seen as a breakthrough in school openness, promising students and parents the kind of insider view into school quality that they'd never been privy to before.

If a school's mission is to educate, then what better way to audit the performance of a school than to track the ultimate measure of that education?

The problems here are many.

Firstly, where death may be the ultimate indicator of poor health, test scores are far from being the ultimate indicator of good learning.

Secondly, Youngme Moon refers to how the act of measurement changes the behaviour of the thing being measured. She is directly referring to what we know as Campbell's Law (I like to refer to this as High Stakes Testing's Kryptonite). Campbell's Law says:


Campbell's law stipulates that "the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it was intended to monitor. Campbell warned us of the inevitable problems associated with undue eight and emphasis on a single indicator for monitoring complex social phenomena. In effect, he warned us about the high-stakes testing program that is part and parcel of No Child Left Behind.
Because the very people who are in the system are corrupted by alluring carrots and threatening sticks, it's no accident that high stake measurements skew reality.

Like the hospitals who might turn away the sickest patients in an attempt to improve its mortality rate - schools might turn away or alienate the weakest students in an attempt to improve their test scores.

Current day accountability measures attempt to ensure at least a basic level of standards are upheld; ironically, it may be those very same accountability measures that constantly drop the bar to the lowest common standard. Or as Youngme Moon puts it, this kind of accountability encourages a herdlike regression toward the mean.

Schools aren't getting better, they are simply becoming more like each other.

We are drowning in standardized mediocrity.

All this lends itself well to understanding what educational psychologist Gerald Bracey said:

There is a growing technology of testing that permits us now to do in nanoseconds things that we shouldn't be doing at all.

High stakes testing encourages schools to see struggling students as an albatross - they are seen as clientel who should be avoided. Like the hospital who turn away the sickest patients - this is a disturbingly chilling vision of public education.

4 comments:

  1. This reminds me of the Australian entrepreneur in Drive that gave his employees 20% of their time to do whatever they liked. The CFO freaked out, calculating that if employees chose to abuse this privelege, they could lose millions. The CEO refused to monitor his employees though, knowing the very observation would erode trust and ironically push employees to start abusing the system. So he ignored his CFO's fears, employees were never held accountable for their 20% time, and profits continued to soar.

    For so many people, it's just too damn scary to have faith in their fellow (wo)man, no matter how much data you give them to show that it works.

    ReplyDelete
  2. Joe,

    enjoyed your blog. Alfie Kohn says the results of testing can be predicted by the number of years of post secondary education of the mother of the child. I teach at a community school in Saskatoon and face the possiblity of test scores being shared in the newspaper.
    Given that, what I find equally alarming is that schools can't seem to figure out what and how they need to change, to improve the quality of our product.

    Tim Comfort

    http://classblogmeister.com/blog.php?blogger_id=253244
    http://educationalchangemanagement.blogspot.com/2010/05/why-educational-change-management.html

    ReplyDelete
  3. Amen and amen. That last sentence is brilliant.

    -Cheryl

    ReplyDelete
  4. Thanks for introducing Youngme Moon to me. It might be interesting to juxtapose a contrary argument some time. I say this because I am drawn into the clarity of the ideas you share and wonder if I am maintaining a balance.

    Tim Comfort closes by expressing his frustration with the extremity of the problem. It is a tangle. After many years I have resigned myself to this reality. There is no straight path and certainly no single path to improved learning. There is only the next group, the next individual, and the next teachable moment. The bean-counters have absolutely no idea.

    This morning I watched a hyperactive uber-geek nine-year-old volunteer to pair up with a timid socially awkward ten-year-old who has never been in a regular classroom. He drew the shy boy into the activity and helped him take the first steps into learning. No sterile curriculum benchmark or clever test will ever measure that moment adequately. It makes me weep.

    ReplyDelete