Unquestioned assumptions about standardized testing

In The Myths of Standardized Tests, the authors write about why it is necessary for us to stop for a moment and think about what standardized tests are capable of telling us and what we really want to know. The premise behind their book is that these are not always the same things. Far too many assumptions underlie standardized testing. 

Standardized testing is a topic most people do not reflect on because we lack the time, effort and expertise. Nonetheless, here are some of their questions to encourage some serious thought:

• Have you ever thought about how well students' knowledge and skills can be assessed by the limited sample of content included in a forty-five question test? What does a score on that test tell you about the vast range of content that simply can't be included?  

• Have you ever talked about the high achievement at a particular school when all you really knew about the school was the average test scores of its students?  

• Have you ever argued -- or heard someone argue -- that what we need is objective information about student achievement? For most people that word objective used in a school context automatically means standardized test scores and very little else. 

• Have you or your school system every handed out punishments or rewards to schools, to teachers, or to individual children based on their test scores? How motivational are such practices? 

• Have you ever thought that improvement in scores on "high stakes" tests is a sound indicator of improvement in learning? 

• Have you ever wondered about whether the tests have an effect on the curriculum and on classroom life? Have you ever questioned what's left out to make time for the tests themselves and for the often extensive preparation for them? 

• Have you ever given more weight to an "indirect" measure (a standardized test score) of student achievement than to a "direct" assessment of achievement? Direct assessments range from judgements teachers make to your own reading of your children's work to the response of those who attend a school performance or a school open house. 

• Have you ever thought that moving to a district or attendance area with high test scores would mean high achievement and success in life for your children? How well do standardized tests forecast future success in school, of course, but also throughout life?

Standardized testing will never be confused with something fun to think about or do, but neither is cancer. Until good people take the time and effort to become knowledgable and skillful, bad things will continue to happen.

Test Score Ambiguity

The problem with using test scores to tell us something about a student, teacher, school or community is that there are far too many variables.

Here's a question from the book The Myths of Standardized Tests:

If 85 percent or more the students in your child's classroom or school meet or exceed the proficiency standards, that means: 

a) your child has an exemplary teacher.
b) your school has an exemplary principal.
c) both a and b.
d) your school community is wealthier than average.
e) all, any combination, or none of the above.

While it may be true that inside the classroom the quality of the teacher has the greatest influence on student learning, the rest of the world outside of the classroom is much larger -- which is why testing experts like Harvard's Daniel Koretz warn:

A great many things other than the quality of schools influence educational achievement, and the impact of these noneducational factors can be huge... 

People routinely misinterpret differences in test scores, commonly attributing more to quality of education than they ought... 

Trends in scores over time, whether down or up, are often influenced by social factors and, in the case of seeming improvements, by inappropriate teaching to the test. Not all low scoring schools offer as weak an educational program as their scores might suggest. By the same token, if your neighborhood schools have high scores, that may mean less about the quality of their programs than you'd like. 

 The point to be taken here is that when we ask tests to be a window into the quality and quantity of student learning AND an educator's teaching, we are asking test scores to do something they can never do.

Finding what we look for

In their book The Myths of Standardized Tests, Phillip Harris, Bruce Smith and Joan Harris tell this story:

"What are you doing?" a helpful passerby asks.
"Looking for my car keys," answers the drunk.
"Did you drop them somewhere around here?"
"I don't think so," replies the drunk.
"Then why look here? the puzzled would-be helper wonders.
"It's the only place where there's any light."

What we find is largely dependent on where we look. The more we tighten our focus on highly prescribed curriculums that are enforced by test and punish standardized exams the more we miss. Ironically, an intense focus requires a kind of tunnel vision that blinds us to the wider consequences of our decisions.

Here's what I mean:

Before you read further, you might want to try out this selective attention experiment.



One of the designers of the experiment, Dr. Daniel Simons, explains what he's learned from conducting this experiment around the world:

We experience far less of our visual world than we think we do. We feel like we are going to take in what's around us. But we don't. We pay attention to what we are told to attend to, or what we're looking for, or what we already know. Top-down factors play a big role. Fashion designers will notice clothes. Engineers will notice mechanics. But what we see is amazingly limited.

In her book Willful Blindness, Margaret Heffernan puts it this way:

We see what we expect to see, what we're looking for. And we can't see all that much. 

And when Heffernan asked Simons if some people see more than others here was his response:

There is really limited evidence for that. People who are experienced basketball players are slightly better at seeing what's happening in the video - but that's probably because they're more accustomed to watching passes; it isn't so hard for them to read what's going on. You can train yourself to focus on more than one spot. You might improve your eye muscles somewhat. But the limits are pretty fixed. There's a physical and an evolutionary barrier. You can't change the limits of your mind.

The point to be taken here for educators is that our attention and vision is biologically limited, and the more time and effort we spend collecting and analyzing test scores, the less time and effort we can expend looking at things that are never found on tests like creativity, perseverance, empathy, resourcefulness and work ethic. In life, there's always too much data. The trick is knowing which to collect and which to let go. The same is true with learning. And unfortunately, today's accountability regimes are encouraging educators to become slaves to the wrong sort of data.

Here's Simons:

For the human brain attention is a zero-sum game: If we pay more attention to one place, object, or event, we necessarily pay less attention to others.

The more we focus on data-driven decisions based on measurable outcomes, the less we attend to educating the whole-child. This might look something like this:

"What are you doing?" a helpful passerby asks.
"Looking for learning," answers the teacher.
"Is there learning in that test?"
"I'm not sure," replies the teacher.
"Then why look here? the puzzled would-be helper wonders.
"This is the easiest place to look."

Education for all

Alfie Kohn writes in his article Debunking the Case for National Standards:

Are all kids entitled to a great education? Of course. But that doesn’t mean all kids should get the same education. High standards don’t require common standards. Uniformity is not the same thing as excellence – or equity. (In fact, one-size-fits-all demands may offer the illusion of fairness, setting back the cause of genuine equity.) To acknowledge these simple truths is to watch the rationale for national standards – or uniform state standards -- collapse into a heap of intellectual rubble.

Stanford University's Elliot Eisner adds:

The kind of schools we need would not hold as an ideal that all students get to the same destinations at the same time. They would embrace the idea that good schools increase the variance in student performance and at the same time escalate the mean.

In The Myth of Standardized Tests, the authors paraphrase Eisner:

That's a professional way of saying that we ought to  become more different, rather than more alike, as we grow to physical and intellectual maturity. More different, yet we all grow.

Mara Sapon-Shevin, the author of Widening the Circle, adds:

What we teach people about human variation as children will have a profound effect on their understanding of difference in the future and their abilities to connect and relate to people who are different from them.

Mara Sapon-Shevin goes on to say what I believe is to be the whole point of this post:

It seems obvious to say that we can understand and value differences only if we are surrounded by them.

Teaching to the test is malpractice

"For every complex problem, there is an answer that is clear, simple - and wrong."

H.L Mencken

Testing is not teaching.

If you want proof of why, you need not look any further than this:

Learning is messy. Real learning is really messy. Testing is, if nothing else, orderly. See the problem?

Talk to any test-maker or psychometrician, and they'll tell you the tests were never devised to make large sweeping, all-encompassing inferences. Even those who speak in favor of using test scores in moderation in low-stakes contexts understand that tests are merely a small sample of a much larger domain that we want to know about, and that great caution must be made in making inferences based on these tests.

So even if we lived in a fictional world where test scores were valid, reliable and objective enough to make sweeping inferences about student achievement, we would have to still admit that the tests can't measure everything.

In their book, The Myths of Standardized Tests, Phillip Harris, Bruce Smith and Joan Harris aptly describe the danger of teaching to any test:

You'll often hear someone say that a good test is one that teachers should be pleased to teach to. But this proposition concerns us. When it comes to whether teaching to the test is a worthy goal, we don't worry so much about the items on the test. If we don't overinterpret what the tests are capable of telling us, we shouldn't do too much damage. However, we think that a far more important issue is almost always overlooked in policy discussion: what's not on the test.

So what kinds of things aren't on tests?

Gerald Bracey offers the start of a list in his book Setting the Record Straight:

• creativity
• critical thinking
• resilience
• motivation
• persistence
• curiosity
• endurance
• reliability
• enthusiasm
• empathy
• self-awareness
• self-discipline
• leadership
• civic-mindedness
• courage
• compassion
• resourcefulness
• sense of beauty
• sense of wonder
• honesty
• integrity

If these are some of the things tests don't even try to measure, perhaps the use of test scores as a solution for the complex problem of measuring learning is in fact clearly and simply wrong.