Standardized test scores are like a broken clock

Have you ever heard someone use standardized test scores to judge schools?

The Alberta Government recently released an information bulletin that boasted Alberta student performance results continue to rise:

The overall percentage of students who attained a standard of excellence on Grade 3, 6, and 9 provincial achievement tests (PATs) increased to 20.2 per cent from 19.5 per cent in the previous year. The percentage of students who met the acceptable standard also rose slightly to 75.5 per cent from 75.2 per cent. One of the highlights of the results is the percentage of students who achieved the standard of excellence in Science 6 and Science 9.

Many Albertans might take these standardized test score results as prima facia evidence that things are well. Many Albertans may be satisfied with this information and confidently move on with their regularly scheduled day, thinking that Alberta schools are not only doing well, but they are improving.

What if we are wrong? What if these scores are giving us false confidence? What if standardized test scores aren't telling us what we think they are telling us?

When some Albertans boasted about these results on Twitter, I responded with:

Assessing an education system via standardized test scores is like assessing a car by kicking the tires.

Some challenged me by asking:

Wouldn't the analogy be, "like assessing a car by comparing its gas mileage relative to motor size and tank capacity?"

My response: 

No. 

The assumption made by this analogy is that we think we know what standardized test scores tell us: we assume these scores are our window into the schools -- therefore we assume we can use these scores to judge the quality of teaching and learning that goes on in a school.

But what if these unquestioned assumptions about standardized testing are wrong?

Seth Godin writes:

The worst kind of clock... is a clock that's wrong. Randomly fast or slow. 

If we know exactly how much it's wrong, then it's not so bad. 

If there's no clock, we go seeking the right time. But a wrong clock? We're going to be tempted to accept what it tells us. 

What are you measuring? Keeping track of the wrong data, or reading it wrong is worse than not keeping track at all.

Standardized test scores are like a broken clock because we assume that these scores tell us what we need to know about our schools -- we assume that these scores reflect teaching and learning and therefore assume that if the numbers are rising that must be a good thing.

But what if this is misguided? What if our reliance on standardized tests to judge the quality of the teaching and learning in schools is like relying on a broken clock for time?

Consider this:

• Standardized test scores are a remarkable way of assessing the socioeconomic status of students and their families. Study after study has shown that out-of-school factors account for an overwhelming proportion of the variances in scores. That means that standardized tests tend to tell us more about what kids bring to school than what they do at school. Here's a Canadian example and an American example.

• There is research that suggests there is a statistical association between high scores on standardized tests and relatively shallow thinking.

• Campbell's Law tells us that the more standardized test scores are used for making decisions and judging schools, the more subject the scores will be to corruption and the more apt it will be to distort and corrupt the teaching and learning it was intended to monitor. This includes cheating scandals, teaching to the test and narrowing of educational opportunities.

• Standardized tests tend to measure what is easily measurable, which turns out to be what matters the least. There is a big difference between measuring what we value and valuing what we measure. When we narrow what matters to what can be measured by a standardized test, we fall victim to the MacNamara Fallacy which basically looks like this: (1) Measure whatever can be easily measured on a standardized test. (2) Disregard whatever can't be easily measured or given an arbitrary quantitate value. (3) Presume that what can't be measured easily isn't important. (4) Say what can't be easily measured doesn't even exist.

• There is research that suggests that when teachers are held accountable for their students' standardized test scores, they tend to become so controlling in their teaching style that the quality of students' performance actually declines.

To fully grasp why this is true, there's a lot to know about the arcane underpinnings of standardized tests; however, testing guru Daniel Koretz gives us a single principle that summarizes what we need to know:

Never treat a test score as a synonym for what children have learned or what teachers have taught.

Again, this too can be true for lots of reasons, but Alfie Kohn gives us a single principle that summarizes what we need to know:

A right answer on a test does not necessarily indicate understanding and a wrong answer does not necessarily indicate a lack of understanding.

I would imagine there are times when standardized test scores might reflect the teaching and learning that goes on in a school, but remember, even a broken clock is right twice a day.

Standardized tests look good from afar but are far from good at reflecting what matters most when it comes to teaching and learning. The closer you look at standardized tests, the more you realize that their utility and convenience comes at an alarming and unacceptable cost. Ask yourself if what we're learning from standardized tests is worth the price.

I would rather no information - no data - nothing! than the grossly misleading and misused data that is extracted from standardized testing. As long as the public is fed standardized test scores, we will be tempted to accept what they tell us -- but if the public had no information about their schools, they would be forced to seek it out which might lead more people to actually step foot in their local schools.