Ranking and Ruining our Schools

When it comes to the Fraser Institute's Schools Rankings, there is little disagreement between the Alberta Government, the Alberta School Board Association and the Alberta Teachers' Association. Talk to a politician, school board trustee or teacher in Alberta and you are likely to hear why it is wrong to rank and sort schools via standardized test scores. In fact, in 2001, the Alberta Government, Alberta School Boards' Association and the Alberta Teachers' Association released a joint statement that said:

Alberta Learning, the Alberta School Boards Association and the Alberta Teachers' Association do not support comparing schools exclusively on the basis of test scores because this provides an incomplete picture of the education provided in any given school. As groups committed to education, we believe strongly in ensuring that parents and students are provided good and complete information as they make the important decisions about which school to attend—how well students do on achievement tests is just one piece of the puzzle.

Years ago, Alberta's then Education Minister Gene Zwozdesky said that it's, "patently unfair to make comparisons between schools based on scores," and last year, Alberta's then Education Minister Thomas Lukaszuk said, "The Provincial Achievement Tests were suppose to inform the province about how well the curriculum is taught, but there are third parties who use the results to rank and sort and blame and shame schools."

If there is so much agreement over why it is wrong to rank and sort schools via standardized test scores, why is the Fraser Institute allowed to continue misusing test scores to mislead the public with their rankings?

It's true, the Fraser Institute is not doing anything illegal, after all, the Provincial Achievement Test scores are public information. But just because something isn't wrong doesn't make it right. 

The Alberta Teachers' Association has been an outspoken critic of standardized tests and has fuelled the argument for more authentic alternatives to public assurance. 

What is the Alberta Government and Alberta School Board Association doing to make sure schools like Braemar are not victimized by school rankings?

Why does the Alberta Government and Alberta School Board Association pursue policies that help third party organizations rank and sort our schools? The silence you hear from the Alberta Government and the Alberta School Board Association is a combination of assent for the rankings and a betrayal to our children.

To paraphrase Edmund Burke, all that is necessary for the misuse of test scores to prevail is for good people to do nothing.

What do standardized test scores tell us?

Today I am going to continue my critique of the Fraser Institute's Report Card on Alberta's High Schools for 2011.

Consider this chart that I created based on information from the Fraser Report:

Here are some interesting details:

• 5 out of the top 20 schools have reported 0% special needs with the highest being 19%. Every single school in the bottom 20 reported a special needs population with the least being 4.9% and the most being 100%.
• 12 out of the top 20 schools have an average parent income over $100,000 and 6 of them were over $200,000. In the bottom 20, not one school has an average parent income over $100,000 while half are below $60,000.
• There are outliers. Bawlf is the only school in the top 20 with an average parent income below $50,000, and there are three schools in the bottom 20 who have an average parent income over $90,000; however, two of those three schools report that over 20% of their population is special needs.

Those in favor of ranking schools via their standardized test scores like to say that it provides parents with the information they need to choose a school for their children. At first glance this looks like it makes a lot of sense -- many people see standardized test scores as the public's window into the quality of our schools. But what if standardized test scores aren't telling us what we think they are telling us? What if standardized test scores tell us less about in-school factors and more about out-of-school factors? In fact, this is exactly the case. Socio-economic status is by far the strongest predictor of student performance on standardized tests.

In Alfie Kohn's book The Case Against Standardized Testing, Kohn explains what standardized testing  really tells us:

The main thing they tell us is how big the students' houses are. Research has repeatedly found that the amount of poverty in the communities where schools are located, along with other variables having nothing to do with what happens in classrooms, accounts for the great majority of the difference in test scores from one area to the next. To that extent, tests are simply not a valid measure of school effectiveness. (Indeed, one educator suggested that we could save everyone a lot of time and money by eliminating standardized tests and just asking a single question: "How much money does your mom make? ... OK, you're on the bottom.") Only someone ignorant or dishonest would present a ranking of schools' test results as though it told us about the quality of teaching that went on in those schools when, in fact, it primarily tells us about socio-economic status and available resources. Of course, knowing what really determines the score makes it impossible to defend the practice of using them as the basis for high-stakes decisions.

When some hear the argument that poverty matters, they like to declare that poverty isn't destiny and that socio-economic status isn't everything. Some will say that within a given school, a group of students of the same status will have variations in the scores. To this Kohn replies:

Sure. And among people who smoke three packs of cigarettes a day, there are going to be variations in lung cancer rates. but that doesn't change the fact that smoking is the factor most powerfully associated with lung cancer.

In Edmonton, Todd Rogers from the University of Alberta conducted research on the variables that affect student performance on Alberta's Provincial Achievement Tests. Rogers found that "by far, the strongest predictor of student performance on achievement tests is socio-economic status (SES)."

In Calgary, Hugh Lytton and Michael Pyryt came to similar conclusions: "Social class factors explain about 45 per cent of the variation in achievement test results. The correlation between income level and achievement test scores is very strong."

Both studies were summarized by the Alberta Teachers' Association News in 1997.

In his book Measuring Up: What Educational Testing Really Tells Us, Daniel Koretz writes about a friend of his that ran a large testing program who often received calls from parents asking him for how they could use standardized test scores to select the best school for their children. Often these phone calls were disappointing for parents because they wanted a method that was simple and free from ambiguity and complexity. Koretz's friend shared an example of when a parent simply wanted a list of the schools with the highest test scores. After trying to explain that test scores shouldn't really be used that way, Koretz's friend lost his patience and told the parent, "If all you want is high average test-scores, tell your realtor that you want to buy into the highest-income neighbourhood you can manage. That will buy you the highest average score you can afford."

Real accountability is about transparency but there is nothing transparent about how standardized testing reduces learning to the convenience of a number or a rank. We are mistakenly led to believe that standardized test scores tell us about school quality when really it is an echo-chamber for affluence and opportunity. Mark Twain may have summarized all this up nicely when he said:

It ain't what you don't know that gets you in trouble. It's what you know for sure that just ain't so.

Thank God for standardized test scores

Because of the Fraser Report and standardized testing, I now know the name of the worst school in Edmonton, Alberta, Canada, and third worst school in the entire province.

The name of that school is Braemar.

So what's wrong with Braemar?

They have low standardized test scores.

Things are so bad at Braemar that The Fraser Report gave them a ZERO out of 10 in comparison to other Edmonton schools and ranked them 273 out of 276 in Alberta.

Oh by the way, 100% of the students who attend Braemar are special needs and are either a pregnant or parenting teen ranging from 13 to 19 years of age. That's right Braemar School, which is championed by Edmonton Public Schools, is charged with the sole purpose of providing an education in a supportive environment exclusively for pregnant and parenting teens.

However, don't let students like Kayley Foerster who was so determined to get her diploma that she wrote it while in labour with her son Peighton distract you from what's really important here -- Braemar has low standardized test scores.

It is critically important to not let the messy and human context of Braemar's students get in the way of accountability. As an Albertan and a tax payer we all have the right to know about Braemer's poor performance for two reasons. One, it helps parents choose; two, it helps schools improve.

Thanks to the Fraser Report and standardized test scores, Albertans are equipped with the information they need to make damn sure that if their teenage daughters get pregnant, they will not attend Braemar; and all those pregnant or parenting teens that feel so frightened and alone will know not to bother enlisting Braemar's help in getting extra support in the form of day care, financial advice and assistance, medical expertise and help with career choices and post-secondary admittance.

And now that the teaching staff at Braemar are aware that the rest of Alberta knows of their low scores and abysmal rank, they can redouble their efforts to improve their pregnant and parenting students' test scores by working with schools who have higher standardized test scores and rank. The bad news is that 6 out of the Fraser Reports top 10 schools have reported 0% or not applicable for their special needs population, and only 1 school has more than 10%.

But there is good news, like Braemer School, Rundle Academy is only one of two other schools that also report that 100% of their students are special needs. As an Albertan and a tax payer, I demand to know when the Braemer staff plan on visiting Rundle Academy or when staff from Rundle Academy will visit Braemer so that I know that they are collaborating in an effort to make Braemer better. After all, this is one of the very reasons we have The Fraser Report and standardized tests -- so public schools like Braemer and private schools like Rundle work together.

Rundle Academy is a private school that provides specialized education for students with diagnosed learning disabilities, and like Braemar, Rundle Academy also has a selection process. While Braemar's admissions policy only accepts pregnant and parenting teens, Rundle Academy also has an application procedure that includes submitting:

• Completed application form
• Copy of the student's birth certificate
• Recent photograph of the student
• Last two years of student's final report card where applicable
• Most recent Provincial Achievement test results where applicable
• $100 non-refundable application fee
• Copy of the student's psych-educational evaluation, if applicable
• SSAT results or the date on which your child will write the exam

Oh by the way, grade 11 and 12 offered at Rundle are limited to university prep courses; but the good news is that further assessment is usually waived if the students' academic records indicate that they have the necessary potential and academic background to be successful. After all, Rundle Academy takes pride in only accepting those students who are a good fit for their school.

Oh and one more thing, tuition for junior and senior high students at Rundle Academy ranges from $11,500 to $13,400 per year. Braemar, on the other hand, is a public school so there's no tuition.

Thank God for the Fraser Report and standardized test scores because without them, schools like Braemer and Rundle Academy would never work together to make Alberta schools a better place for kids...

Let's pause for a second. Does any of this bother you? Because it's starting to bother me...

Does anyone think for one-second that the context of the students' needs or the educators' teaching assignments at Braemar have much to do with the kinds of services offered or not offered at Rundle Academy?

Can you see how two schools with 100% special needs might have the responsibility of educating two very different groups of students? How does it help the citizens and tax payers of Alberta to compare a private school that has an admission's policy that determines if students are a good fit if they have academic records and background that indicate potential success and charge a tuition fee over $10,000 with a public school that educates pregnant and parenting teens for free?

And even if Braemar did have something in common with Rundle Academy, does anyone believe a private school in Calgary would utilize their time, effort and resources to help a public school in Edmonton?

I have absolutely no idea how this misuse of data could be sold as a public service.

People who need standardized test scores like to say that they offer accountability to the public. But this is asinine. Accountability is really about transparency, but there is absolutely NOTHING transparent about ranking and rating schools by their standardized test scores. Reducing the magnificently messy stuff that schools do everyday to a test score or grade conceals far more than it ever reveals and therefore is not transparent or accountable -- it's merely convenient.

If all you know about a school or student is their standardized test scores, then you don't know very much. Let me be crystal clear, I've never been to Braemar School or Rundle Academy, but I bet the teachers there are both working very hard to do right by their students and provide them with the best education they can. I won't presume to pass judgment on either Braemar or Rundle. And that is exactly what is so wrong about standardized test scores and The Fraser Report -- with them, people who have never stepped foot in either school can claim to know enough about them to rate and rank them. This is irresponsible and flat out unacceptable.

As tragic as it is that Braemar School has been so wrongly judged for its low standardized test scores, remember that every school suffers from this primitive form of information and misuse of data. Governments that hand over standardized test scores to third party organizations so that they can pit schools against one another by ranking and rating them is the equivalent of handing someone who is drunk the keys to their car, and the school board associations that support the government's release of this data are accomplices to all this misuse. Just as institutions that sell their standardized test score snake oil as a public service should be held accountable for deceiving the public so too should the governments that provide the data and the school board associations that support it.

Michael Thomas, a co-author of The Fraser Report's Report Card says that "no one has ever shown me how it's damaging a school." I've gone through their report card and I noticed that you can contribute to the development of the Report Card by contacting Michael Thomas at 416.363.6575 with your suggestions, comments and criticisms. 

After hearing about how Braemar has been treated by the Fraser Report, I am going to call Michael right now and tell him why I think their misuse of standardized test scores is damaging to schools. Will you?

* A few edits were made to clarify confusion between Rundle College and Rundle Academy.