What do standardized test scores tell us?

Today I am going to continue my critique of the Fraser Institute's Report Card on Alberta's High Schools for 2011.

Consider this chart that I created based on information from the Fraser Report:

Here are some interesting details:

• 5 out of the top 20 schools have reported 0% special needs with the highest being 19%. Every single school in the bottom 20 reported a special needs population with the least being 4.9% and the most being 100%.
• 12 out of the top 20 schools have an average parent income over $100,000 and 6 of them were over $200,000. In the bottom 20, not one school has an average parent income over $100,000 while half are below $60,000.
• There are outliers. Bawlf is the only school in the top 20 with an average parent income below $50,000, and there are three schools in the bottom 20 who have an average parent income over $90,000; however, two of those three schools report that over 20% of their population is special needs.

Those in favor of ranking schools via their standardized test scores like to say that it provides parents with the information they need to choose a school for their children. At first glance this looks like it makes a lot of sense -- many people see standardized test scores as the public's window into the quality of our schools. But what if standardized test scores aren't telling us what we think they are telling us? What if standardized test scores tell us less about in-school factors and more about out-of-school factors? In fact, this is exactly the case. Socio-economic status is by far the strongest predictor of student performance on standardized tests.

In Alfie Kohn's book The Case Against Standardized Testing, Kohn explains what standardized testing  really tells us:

The main thing they tell us is how big the students' houses are. Research has repeatedly found that the amount of poverty in the communities where schools are located, along with other variables having nothing to do with what happens in classrooms, accounts for the great majority of the difference in test scores from one area to the next. To that extent, tests are simply not a valid measure of school effectiveness. (Indeed, one educator suggested that we could save everyone a lot of time and money by eliminating standardized tests and just asking a single question: "How much money does your mom make? ... OK, you're on the bottom.") Only someone ignorant or dishonest would present a ranking of schools' test results as though it told us about the quality of teaching that went on in those schools when, in fact, it primarily tells us about socio-economic status and available resources. Of course, knowing what really determines the score makes it impossible to defend the practice of using them as the basis for high-stakes decisions.

When some hear the argument that poverty matters, they like to declare that poverty isn't destiny and that socio-economic status isn't everything. Some will say that within a given school, a group of students of the same status will have variations in the scores. To this Kohn replies:

Sure. And among people who smoke three packs of cigarettes a day, there are going to be variations in lung cancer rates. but that doesn't change the fact that smoking is the factor most powerfully associated with lung cancer.

In Edmonton, Todd Rogers from the University of Alberta conducted research on the variables that affect student performance on Alberta's Provincial Achievement Tests. Rogers found that "by far, the strongest predictor of student performance on achievement tests is socio-economic status (SES)."

In Calgary, Hugh Lytton and Michael Pyryt came to similar conclusions: "Social class factors explain about 45 per cent of the variation in achievement test results. The correlation between income level and achievement test scores is very strong."

Both studies were summarized by the Alberta Teachers' Association News in 1997.

In his book Measuring Up: What Educational Testing Really Tells Us, Daniel Koretz writes about a friend of his that ran a large testing program who often received calls from parents asking him for how they could use standardized test scores to select the best school for their children. Often these phone calls were disappointing for parents because they wanted a method that was simple and free from ambiguity and complexity. Koretz's friend shared an example of when a parent simply wanted a list of the schools with the highest test scores. After trying to explain that test scores shouldn't really be used that way, Koretz's friend lost his patience and told the parent, "If all you want is high average test-scores, tell your realtor that you want to buy into the highest-income neighbourhood you can manage. That will buy you the highest average score you can afford."

Real accountability is about transparency but there is nothing transparent about how standardized testing reduces learning to the convenience of a number or a rank. We are mistakenly led to believe that standardized test scores tell us about school quality when really it is an echo-chamber for affluence and opportunity. Mark Twain may have summarized all this up nicely when he said:

It ain't what you don't know that gets you in trouble. It's what you know for sure that just ain't so.