Ranking and Ruining our Schools

When it comes to the Fraser Institute's Schools Rankings, there is little disagreement between the Alberta Government, the Alberta School Board Association and the Alberta Teachers' Association. Talk to a politician, school board trustee or teacher in Alberta and you are likely to hear why it is wrong to rank and sort schools via standardized test scores. In fact, in 2001, the Alberta Government, Alberta School Boards' Association and the Alberta Teachers' Association released a joint statement that said:

Alberta Learning, the Alberta School Boards Association and the Alberta Teachers' Association do not support comparing schools exclusively on the basis of test scores because this provides an incomplete picture of the education provided in any given school. As groups committed to education, we believe strongly in ensuring that parents and students are provided good and complete information as they make the important decisions about which school to attend—how well students do on achievement tests is just one piece of the puzzle.

Years ago, Alberta's then Education Minister Gene Zwozdesky said that it's, "patently unfair to make comparisons between schools based on scores," and last year, Alberta's then Education Minister Thomas Lukaszuk said, "The Provincial Achievement Tests were suppose to inform the province about how well the curriculum is taught, but there are third parties who use the results to rank and sort and blame and shame schools."

If there is so much agreement over why it is wrong to rank and sort schools via standardized test scores, why is the Fraser Institute allowed to continue misusing test scores to mislead the public with their rankings?

It's true, the Fraser Institute is not doing anything illegal, after all, the Provincial Achievement Test scores are public information. But just because something isn't wrong doesn't make it right. 

The Alberta Teachers' Association has been an outspoken critic of standardized tests and has fuelled the argument for more authentic alternatives to public assurance. 

What is the Alberta Government and Alberta School Board Association doing to make sure schools like Braemar are not victimized by school rankings?

Why does the Alberta Government and Alberta School Board Association pursue policies that help third party organizations rank and sort our schools? The silence you hear from the Alberta Government and the Alberta School Board Association is a combination of assent for the rankings and a betrayal to our children.

To paraphrase Edmund Burke, all that is necessary for the misuse of test scores to prevail is for good people to do nothing.

What do standardized test scores tell us?

Today I am going to continue my critique of the Fraser Institute's Report Card on Alberta's High Schools for 2011.

Consider this chart that I created based on information from the Fraser Report:

Here are some interesting details:

• 5 out of the top 20 schools have reported 0% special needs with the highest being 19%. Every single school in the bottom 20 reported a special needs population with the least being 4.9% and the most being 100%.
• 12 out of the top 20 schools have an average parent income over $100,000 and 6 of them were over $200,000. In the bottom 20, not one school has an average parent income over $100,000 while half are below $60,000.
• There are outliers. Bawlf is the only school in the top 20 with an average parent income below $50,000, and there are three schools in the bottom 20 who have an average parent income over $90,000; however, two of those three schools report that over 20% of their population is special needs.

Those in favor of ranking schools via their standardized test scores like to say that it provides parents with the information they need to choose a school for their children. At first glance this looks like it makes a lot of sense -- many people see standardized test scores as the public's window into the quality of our schools. But what if standardized test scores aren't telling us what we think they are telling us? What if standardized test scores tell us less about in-school factors and more about out-of-school factors? In fact, this is exactly the case. Socio-economic status is by far the strongest predictor of student performance on standardized tests.

In Alfie Kohn's book The Case Against Standardized Testing, Kohn explains what standardized testing  really tells us:

The main thing they tell us is how big the students' houses are. Research has repeatedly found that the amount of poverty in the communities where schools are located, along with other variables having nothing to do with what happens in classrooms, accounts for the great majority of the difference in test scores from one area to the next. To that extent, tests are simply not a valid measure of school effectiveness. (Indeed, one educator suggested that we could save everyone a lot of time and money by eliminating standardized tests and just asking a single question: "How much money does your mom make? ... OK, you're on the bottom.") Only someone ignorant or dishonest would present a ranking of schools' test results as though it told us about the quality of teaching that went on in those schools when, in fact, it primarily tells us about socio-economic status and available resources. Of course, knowing what really determines the score makes it impossible to defend the practice of using them as the basis for high-stakes decisions.

When some hear the argument that poverty matters, they like to declare that poverty isn't destiny and that socio-economic status isn't everything. Some will say that within a given school, a group of students of the same status will have variations in the scores. To this Kohn replies:

Sure. And among people who smoke three packs of cigarettes a day, there are going to be variations in lung cancer rates. but that doesn't change the fact that smoking is the factor most powerfully associated with lung cancer.

In Edmonton, Todd Rogers from the University of Alberta conducted research on the variables that affect student performance on Alberta's Provincial Achievement Tests. Rogers found that "by far, the strongest predictor of student performance on achievement tests is socio-economic status (SES)."

In Calgary, Hugh Lytton and Michael Pyryt came to similar conclusions: "Social class factors explain about 45 per cent of the variation in achievement test results. The correlation between income level and achievement test scores is very strong."

Both studies were summarized by the Alberta Teachers' Association News in 1997.

In his book Measuring Up: What Educational Testing Really Tells Us, Daniel Koretz writes about a friend of his that ran a large testing program who often received calls from parents asking him for how they could use standardized test scores to select the best school for their children. Often these phone calls were disappointing for parents because they wanted a method that was simple and free from ambiguity and complexity. Koretz's friend shared an example of when a parent simply wanted a list of the schools with the highest test scores. After trying to explain that test scores shouldn't really be used that way, Koretz's friend lost his patience and told the parent, "If all you want is high average test-scores, tell your realtor that you want to buy into the highest-income neighbourhood you can manage. That will buy you the highest average score you can afford."

Real accountability is about transparency but there is nothing transparent about how standardized testing reduces learning to the convenience of a number or a rank. We are mistakenly led to believe that standardized test scores tell us about school quality when really it is an echo-chamber for affluence and opportunity. Mark Twain may have summarized all this up nicely when he said:

It ain't what you don't know that gets you in trouble. It's what you know for sure that just ain't so.

The Folly of Ranking Schools by Test Scores

This was written by Alfie Kohn for the Calgary Herald in February of 2005.

by Alfie Kohn

There’s bad news in today’s Outlook section. It’s not that the test scores listed there are low. It’s that the test scores are listed there at all.

As a rule, the less people know about education – and about exams like the Provincial Achievement Test -- the more willing they are to judge schools (and students) on the basis of such scores. It’s not that standardized test results don’t tell us anything. They’re very accurate measures of the size of the houses near a given school. As many researchers, including Todd Rogers at the University of Alberta, have shown, a school’s test scores can be predicted with reasonable accuracy if one knows the socioeconomic status of its students – and nothing about what happens in its classrooms.

But what if a school’s scores are rising over the years despite having roughly the same kind of students? In that case, the results aren’t meaningless -- they’re worrisome. Better results on the P.A.T. may actually be a bad sign because of what had to be sacrificed in order to make that happen.

Those numbers in the Outlook section do not reflect academic excellence. Indeed, teachers in Alberta and elsewhere often tell me that some of their most impressive students don’t perform well on these exams, whereas other kids get terrific scores just because they’re good at taking tests. The research confirms this: Three studies have found that high standardized test scores often go hand-in-hand with superficial thinking. What’s true of individuals is true of schools. Some of the best may not be at the top of those rankings precisely because they don’t stuff kids full of testable (and forgettable) facts. Rather than slavishly covering the provincial curriculum, teachers in great schools help children to discover ideas. Their students are learning what it means to think like a scientist or an historian, not merely memorizing dates and definitions.

Indeed, in visiting Saskatchewan and Manitoba, I’ve been impressed by the nearly complete absence of standardized tests there. They spend their time and resources on teaching rather than testing, and their schools are better off as a result.

Any attempt to reduce learning to numbers is misguided. Schools just can’t be rated like laundry detergents. As one U.S. educational researcher put it, “Measurable outcomes may be the least significant results of learning.” The damage is compounded when those numbers are test scores. Exams like the P.A.T. tend to measure what matters least.

Worst of all is using these scores not just to rate but to rank, such that the emphasis is on who’s beating whom. This encourages a toxic competition between schools. Educators are less inclined to work together to do a better job at educating all our children because they’re forced to worry about their relative standing.

There are only two explanations for the wretched ritual of publishing these scores. The first is simple ignorance. The second is some sort of political agenda – exactly the kind, come to think of it, pursued by the Fraser Institute, which wants to privatize everything in sight and privilege profit-oriented enterprises over democratic public institutions.

Frankly, if my goal were to make public schools look bad and pave the way for vouchers, I would do exactly what they’ve done: use the language of “accountability” to push for more emphasis on testing -- and set schools against one another.

But a right-wing think tank can’t accomplish this alone. It needs Alberta Learning notonly to force schools to give all those tests but to boil down the results into summary statistics that are ripe for publication. (The State of Connecticut refuses to do this in order to thwart those who would try to reduce schools to a single set of numbers.) Last week, Education Minister Gene Zwozdesky told a CBC interviewer that it’s “patently unfair to make comparisons between schools based on scores.” The question is why he is pursuing policies that facilitate just such comparisons.

The Fraserites also need newspapers like the Herald to do their dirty work for them by publishing these charts. The effects aren’t always easy to see or measure, but every year about this time, Calgary’s schools become a little worse because this paper leads parents and teachers to focus on test scores rather than on more meaningful indicators of quality.

Elsewhere, I’ve written at length about those indicators. Is the focus more on understanding ideas from the inside out (rather than on memorizing facts)? Do students experience their schools as caring communities? Do teachers create democratic classrooms so kids can participate in making decisions? Do children often learn together (rather than alone)? Do parents receive qualitative accounts of kids’ improvement (rather than traditional grades)?

By contrast, the most constructive thing you can do with those pages in today’s Outlook section is use them to wrap fish. The numbers printed there are not only meaningless but dangerously misleading.

One figure, though, is worth looking at: the percentage of “exams not written” at each school. When that number begins to climb, it will mean that Calgary-area parents are realizing that they’ve contributed to the problem by allowing their children to write the exams. When enough of them get organized and decide to boycott the P.A.T.s, Alberta Learning will ask nervously, “What if we gave a test and nobody came?”

That will mean the flourishing of grassroots democracy and a more sensible set of educational priorities. And that, finally, will be something worth reporting.

Fraser Institute: Data mongers

I was reading Student awards cut to foster teamwork in B.C., Alberta when I came across this:

“Competition is a mechanism by which to encourage excellence among students and to equip them with the real-life skills of getting used to the idea that in various aspects of their lives they will be competing and that some will win and some will lose,” said Peter Cowley, an education policy researcher at the Fraser Institute in Vancouver.

It is important to note that the Fraser Institute is to schools as a vulture is to the living. Peter Cowley and his cult of data-mongers feed off pitting schools against schools, and ultimately, children against children.

Just like the ring leader who spurs on the animals and spectators during blood sports, Peter Cowley and the Fraser Institute want nothing more than to perpetuate the zero-sum game that defines competition. Schools duke it out in an attempt to get high scores on bad tests, while The Fraser Instittue profits from the sidelines.

Ken Chapman properly exposes the Fraser Institute for the fools they really are in his post titled Fraser Institute Report Comparing Schools is Educational Folly :

I always dispair at the superficial analysis and misleading inferences from the Fraser Institute when it ranks schools based on standardized test results. When you ask a shallow and silly question you are bound to get a useless answer. As the world gets more complex and informed engaged citizenship becomes more important than ever we need to ensure the skills we teach are those that are essential for this new and emerging world.

Of course the traditional subjects are still important but not as the only things worth teaching and testing. To compare schools and insist that they compete for credibility when private schools can restrict who they enroll and public schools will and must take all comers and to say this is a quality measure is misleading at best. This fundamental reality about enrollment and the socio-economic differences in schools make the Fraser Institute comparison reporting such a disservice. How are students, teachers, parents and the public able to use such selective comparisons when trying to discern if our education system is doing the job it needs to do to prepare the whole student for the changing and emerging reality they will face.

If students and teachers are only ranked and rated on narrow focued standardized test results we only get a bellcurve distribution but no insight as to how well prepared the whole student is for adaptation, resilience, self-sufficiency and survival in a complex interdependent globalized social, environmental, economic and political culture.

The Fraser Institute reports on public education is as helpful as counting the number of nails in a house and presuming that measures its value to those who live in it. It is not even worthy of being taken with a grain of salt. It is beyond useless, it is dangerous.